[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:37:04 dist-test-slave-rmhg krb5kdc[3599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:37:04 dist-test-slave-rmhg krb5kdc[3599](info): set up 2 sockets
Aug 12 02:37:04 dist-test-slave-rmhg krb5kdc[3599](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 02:37:05.663985 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.009s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:37:05 dist-test-slave-rmhg krb5kdc[3599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:37:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:37:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:37:05.801579 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:33729
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:33729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:05.942575 3615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:05.942900 3615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:05.942963 3615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:05.947283 3615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:05.947363 3615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:05.947391 3615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:05.947413 3615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:05.947434 3615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:37:05.953601 3615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:33729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:33729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:05.955135 3615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:05.956558 3615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:37:05.971727 3615 server_base.cc:1034] running on GCE node
W20260812 02:37:05.971808 3630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:05.971930 3623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:05.973356 3625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:05.974319 3615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:05.976257 3615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:05.977663 3615 hybrid_clock.cc:648] HybridClock initialized: now 1786502225977585 us; error 70 us; skew 500 ppm
Aug 12 02:37:05 dist-test-slave-rmhg krb5kdc[3599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502225, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:05.985808 3615 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:37:05.987006 3615 webserver.cc:460] Webserver started at http://127.3.127.190:44091/ using document root <none> and password file <none>
I20260812 02:37:05.988011 3615 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:05.988090 3615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:05.988487 3615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:05.991590 3615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "87f4de57adfd41f6877e164f17dace6d"
format_stamp: "Formatted at 2026-08-12 02:37:05 on dist-test-slave-rmhg"
server_key: "ba6b1c97669d752555c7c413052d0d79"
server_key_iv: "b4b73a15ee63e4afc23836b2e61fb68f"
server_key_version: "encryptionkey@0"
I20260812 02:37:05.992278 3615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "87f4de57adfd41f6877e164f17dace6d"
format_stamp: "Formatted at 2026-08-12 02:37:05 on dist-test-slave-rmhg"
server_key: "ba6b1c97669d752555c7c413052d0d79"
server_key_iv: "b4b73a15ee63e4afc23836b2e61fb68f"
server_key_version: "encryptionkey@0"
I20260812 02:37:05.998449 3615 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 02:37:06.002522 3642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:06.004448 3615 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:37:06.004698 3615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "87f4de57adfd41f6877e164f17dace6d"
format_stamp: "Formatted at 2026-08-12 02:37:05 on dist-test-slave-rmhg"
server_key: "ba6b1c97669d752555c7c413052d0d79"
server_key_iv: "b4b73a15ee63e4afc23836b2e61fb68f"
server_key_version: "encryptionkey@0"
I20260812 02:37:06.004833 3615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:06.020711 3615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:06.024960 3615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:06.025230 3615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:06.042841 3615 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:33729
I20260812 02:37:06.042888 3732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:33729 every 8 connection(s)
I20260812 02:37:06.044243 3615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:37:06.050335 3733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:06.051296 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 3615
I20260812 02:37:06.051460 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:37:06.053110 3582 external_mini_cluster.cc:1428] Setting key 904136bd4cb75f0f7fedee392f072753
I20260812 02:37:06.062767 3733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d: Bootstrap starting.
I20260812 02:37:06.067852 3733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:06.069921 3733 log.cc:824] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:06.073277 3733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d: No bootstrap required, opened a new log
Aug 12 02:37:06 dist-test-slave-rmhg krb5kdc[3599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:06.080050 3733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f4de57adfd41f6877e164f17dace6d" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33729 } }
I20260812 02:37:06.080400 3733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:06.080503 3733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f4de57adfd41f6877e164f17dace6d, State: Initialized, Role: FOLLOWER
I20260812 02:37:06.081483 3733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f4de57adfd41f6877e164f17dace6d" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33729 } }
I20260812 02:37:06.081717 3733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:37:06.081871 3733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:37:06.082016 3733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:06.083568 3733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f4de57adfd41f6877e164f17dace6d" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33729 } }
I20260812 02:37:06.084254 3733 leader_election.cc:302] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87f4de57adfd41f6877e164f17dace6d; no voters:
I20260812 02:37:06.084728 3733 leader_election.cc:288] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:37:06.084903 3743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:06.085289 3743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d [term 1 LEADER]: Becoming Leader. State: Replica: 87f4de57adfd41f6877e164f17dace6d, State: Running, Role: LEADER
I20260812 02:37:06.085803 3743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f4de57adfd41f6877e164f17dace6d" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33729 } }
I20260812 02:37:06.086706 3733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:37:06.087864 3743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87f4de57adfd41f6877e164f17dace6d. Latest consensus state: current_term: 1 leader_uuid: "87f4de57adfd41f6877e164f17dace6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f4de57adfd41f6877e164f17dace6d" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33729 } } }
I20260812 02:37:06.088587 3743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:06.088574 3738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:06.058632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.3.127.190:33729)
0812 02:37:06.059291 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:06.059311 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:06.063276 (+ 3965us) server_negotiation.cc:407] Connection header received
0812 02:37:06.068387 (+ 5111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:06.068408 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:06.068847 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:06.069256 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:06.072311 (+ 3055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:06.073453 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:06.075108 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:06.075664 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:06.075845 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:06.080742 (+ 4897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:06.080794 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:06.080813 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:06.080860 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:06.083503 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:06.084190 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:06.084197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:06.084203 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:06.084367 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:06.084778 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:06.084782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:06.084784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:06.085533 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:06.085730 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:06.086466 (+ 736us) server_negotiation.cc:299] Negotiation successful
0812 02:37:06.086766 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":220,"threads_started":1}
I20260812 02:37:06.091364 3745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87f4de57adfd41f6877e164f17dace6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f4de57adfd41f6877e164f17dace6d" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33729 } } }
I20260812 02:37:06.091498 3745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d [sys.catalog]: This master's current role is: LEADER
W20260812 02:37:06.097558 3765 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:37:06.097712 3765 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:37:06.097864 3757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:37:06.099442 3757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:37:06.108933 3757 catalog_manager.cc:1347] Generated new cluster ID: 425a2d411bf6458782ce45d60e2d3b16
I20260812 02:37:06.109018 3757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:37:06.119961 3757 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:37:06.121423 3757 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:37:06.130771 3757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87f4de57adfd41f6877e164f17dace6d: Generated new TSK 0
I20260812 02:37:06.132055 3757 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
I20260812 02:37:06.179931 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33729
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:06.366472 3775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:06.366771 3775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:06.366819 3775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:06.372262 3775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:06.372362 3775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:06.372454 3775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:37:06.380479 3775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33729
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:06.384413 3775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:06.388537 3775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:06.401966 3783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:06.401906 3786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:06.401887 3784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:06.402355 3775 server_base.cc:1034] running on GCE node
I20260812 02:37:06.402918 3775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:06.403666 3775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:06.404840 3775 hybrid_clock.cc:648] HybridClock initialized: now 1786502226404819 us; error 35 us; skew 500 ppm
Aug 12 02:37:06 dist-test-slave-rmhg krb5kdc[3599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502226, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:06.408829 3775 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:37:06.409931 3775 webserver.cc:460] Webserver started at http://127.3.127.129:38403/ using document root <none> and password file <none>
I20260812 02:37:06.410535 3775 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:06.410620 3775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:06.410861 3775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:06.413300 3775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "b52762573efd4142ba761f0f933e130e"
format_stamp: "Formatted at 2026-08-12 02:37:06 on dist-test-slave-rmhg"
server_key: "b851da767d91c63f7a4ecc0f74a9ddfa"
server_key_iv: "9de282b9025436633ff980060edcdb9e"
server_key_version: "encryptionkey@0"
I20260812 02:37:06.413918 3775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "b52762573efd4142ba761f0f933e130e"
format_stamp: "Formatted at 2026-08-12 02:37:06 on dist-test-slave-rmhg"
server_key: "b851da767d91c63f7a4ecc0f74a9ddfa"
server_key_iv: "9de282b9025436633ff980060edcdb9e"
server_key_version: "encryptionkey@0"
I20260812 02:37:06.418550 3775 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:37:06.421473 3797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:06.423002 3775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 02:37:06.423137 3775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "b52762573efd4142ba761f0f933e130e"
format_stamp: "Formatted at 2026-08-12 02:37:06 on dist-test-slave-rmhg"
server_key: "b851da767d91c63f7a4ecc0f74a9ddfa"
server_key_iv: "9de282b9025436633ff980060edcdb9e"
server_key_version: "encryptionkey@0"
I20260812 02:37:06.423249 3775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:06.452950 3775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:06.457085 3775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:06.457418 3775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:06.458427 3775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:06.459903 3775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:06.459980 3775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:06.460036 3775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:06.460058 3775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:06.483116 3775 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:38495
I20260812 02:37:06.483274 3983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:38495 every 8 connection(s)
I20260812 02:37:06.484203 3775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:37:06.490667 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 3775
I20260812 02:37:06.490797 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:37:06.491050 3582 external_mini_cluster.cc:1428] Setting key 927bf05c57bbec155064e6255e83f7d0
Aug 12 02:37:06 dist-test-slave-rmhg krb5kdc[3599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502226, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.127.130@KRBTEST.COM; defaulting to no policy
I20260812 02:37:06.502313 3738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:06.486849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:37549 (local address 127.3.127.190:33729)
0812 02:37:06.487031 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:06.487035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:06.488397 (+ 1362us) server_negotiation.cc:407] Connection header received
0812 02:37:06.490507 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:06.490511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:06.490576 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:06.490670 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:06.492302 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:06.492835 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:06.494331 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:06.494590 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:06.494686 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:06.498076 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:06.498079 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:06.498082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:06.498128 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:06.499788 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:06.500415 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:06.500418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:06.500420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:06.500558 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:06.500952 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:06.500955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:06.500957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:06.501209 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:06.501305 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:06.501961 (+ 656us) server_negotiation.cc:299] Negotiation successful
0812 02:37:06.502131 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 02:37:06.503331 3989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:06.487341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:33729 (local address 127.3.127.129:37549)
0812 02:37:06.488235 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:06.488289 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:06.490271 (+ 1982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:06.490877 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:06.490885 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:06.491419 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:06.492131 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:06.492149 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:06.492969 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:06.492973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:06.494184 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:06.494197 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:06.494758 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:06.494762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:06.495024 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:06.495846 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:06.495866 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:06.497852 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:06.499939 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:06.499948 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:06.499969 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:06.500289 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:06.500658 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:06.500662 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:06.500664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:06.500841 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:06.501322 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:06.501330 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:06.501675 (+ 345us) client_negotiation.cc:769] Sending connection context
0812 02:37:06.501873 (+ 198us) client_negotiation.cc:240] Negotiation successful
0812 02:37:06.502289 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":245,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.130@KRBTEST.COM" created.
I20260812 02:37:06.505015 3985 heartbeater.cc:347] Connected to a master server at 127.3.127.190:33729
I20260812 02:37:06.505321 3985 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:06.505924 3985 heartbeater.cc:511] Master 127.3.127.190:33729 requested a full tablet report, sending...
I20260812 02:37:06.508445 3669 ts_manager.cc:194] Registered new tserver with Master: b52762573efd4142ba761f0f933e130e (127.3.127.129:38495)
I20260812 02:37:06.510589 3669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:37549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
I20260812 02:37:06.522462 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33729
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:06.690487 3993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:06.690774 3993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:06.690820 3993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:06.694900 3993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:06.694975 3993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:06.695039 3993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:37:06.700863 3993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33729
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:06.702116 3993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:06.703179 3993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:37:06.714136 3993 server_base.cc:1034] running on GCE node
W20260812 02:37:06.714071 4000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:06.714087 4001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:06.714054 4004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:06.714694 3993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:06.715386 3993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:06.716553 3993 hybrid_clock.cc:648] HybridClock initialized: now 1786502226716542 us; error 27 us; skew 500 ppm
Aug 12 02:37:06 dist-test-slave-rmhg krb5kdc[3599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502226, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:06.719455 3993 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:37:06.720371 3993 webserver.cc:460] Webserver started at http://127.3.127.130:45175/ using document root <none> and password file <none>
I20260812 02:37:06.720966 3993 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:06.721030 3993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:06.721287 3993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:06.723479 3993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "36d83c0f5f01469ebd408175db52704d"
format_stamp: "Formatted at 2026-08-12 02:37:06 on dist-test-slave-rmhg"
server_key: "1a24c5f6067948a21825753fb53bace1"
server_key_iv: "d4b65d939365ba1cf57b38502cf1bec9"
server_key_version: "encryptionkey@0"
I20260812 02:37:06.724056 3993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "36d83c0f5f01469ebd408175db52704d"
format_stamp: "Formatted at 2026-08-12 02:37:06 on dist-test-slave-rmhg"
server_key: "1a24c5f6067948a21825753fb53bace1"
server_key_iv: "d4b65d939365ba1cf57b38502cf1bec9"
server_key_version: "encryptionkey@0"
I20260812 02:37:06.728148 3993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 02:37:06.730892 4013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:06.732270 3993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:37:06.732386 3993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "36d83c0f5f01469ebd408175db52704d"
format_stamp: "Formatted at 2026-08-12 02:37:06 on dist-test-slave-rmhg"
server_key: "1a24c5f6067948a21825753fb53bace1"
server_key_iv: "d4b65d939365ba1cf57b38502cf1bec9"
server_key_version: "encryptionkey@0"
I20260812 02:37:06.732511 3993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:06.751009 3993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:06.755276 3993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:06.755499 3993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:06.756366 3993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:06.757511 3993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:06.757565 3993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:06.757609 3993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:06.757627 3993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:06.780388 3993 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:45803
I20260812 02:37:06.780448 4190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:45803 every 8 connection(s)
I20260812 02:37:06.781592 3993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:37:06.781888 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 3993
I20260812 02:37:06.782001 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:37:06.782326 3582 external_mini_cluster.cc:1428] Setting key 300eefdc2c536288320f5f159f1186cb
Aug 12 02:37:06 dist-test-slave-rmhg krb5kdc[3599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502226, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.127.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.131@KRBTEST.COM" created.
I20260812 02:37:06.800194 3738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:06.784325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:45829 (local address 127.3.127.190:33729)
0812 02:37:06.785310 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:06.785313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:06.786077 (+ 764us) server_negotiation.cc:407] Connection header received
0812 02:37:06.788455 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:06.788459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:06.788570 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:06.788707 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:06.790486 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:06.791023 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:06.792588 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:06.792824 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:06.792919 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:06.796186 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:06.796190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:06.796192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:06.796223 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:06.797793 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:06.798441 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:06.798444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:06.798446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:06.798537 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:06.798933 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:06.798936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:06.798938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:06.799165 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:06.799258 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:06.799859 (+ 601us) server_negotiation.cc:299] Negotiation successful
0812 02:37:06.800020 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":892}
I20260812 02:37:06.801088 4196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:06.784937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:33729 (local address 127.3.127.130:45829)
0812 02:37:06.785868 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:06.785918 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:06.788146 (+ 2228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:06.788950 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:06.788959 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:06.789526 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:06.790306 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:06.790326 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:06.791160 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:06.791164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:06.792437 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:06.792451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:06.793001 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:06.793006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:06.793329 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:06.794047 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:06.794066 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:06.795960 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:06.797945 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:06.797952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:06.797968 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:06.798318 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:06.798619 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:06.798622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:06.798624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:06.798830 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:06.799276 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:06.799282 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:06.799572 (+ 290us) client_negotiation.cc:769] Sending connection context
0812 02:37:06.799759 (+ 187us) client_negotiation.cc:240] Negotiation successful
0812 02:37:06.800062 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":352,"threads_started":1}
I20260812 02:37:06.802627 4193 heartbeater.cc:347] Connected to a master server at 127.3.127.190:33729
I20260812 02:37:06.802903 4193 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:06.803511 4193 heartbeater.cc:511] Master 127.3.127.190:33729 requested a full tablet report, sending...
I20260812 02:37:06.804783 3669 ts_manager.cc:194] Registered new tserver with Master: 36d83c0f5f01469ebd408175db52704d (127.3.127.130:45803)
I20260812 02:37:06.805483 3669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:45829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
I20260812 02:37:06.818127 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33729
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:06.994222 4201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:06.994550 4201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:06.994621 4201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:06.999081 4201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:06.999164 4201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:06.999256 4201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:37:07.005511 4201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33729
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:07.006754 4201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:07.007889 4201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:07.020133 4213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:07.020282 4201 server_base.cc:1034] running on GCE node
W20260812 02:37:07.020063 4215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:07.020179 4211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:07.020895 4201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:07.021703 4201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:07.022900 4201 hybrid_clock.cc:648] HybridClock initialized: now 1786502227022876 us; error 36 us; skew 500 ppm
Aug 12 02:37:07 dist-test-slave-rmhg krb5kdc[3599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502227, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:07.026655 4201 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:37:07.027793 4201 webserver.cc:460] Webserver started at http://127.3.127.131:46699/ using document root <none> and password file <none>
I20260812 02:37:07.028436 4201 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:07.028517 4201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:07.028750 4201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:07.031360 4201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "c0134a2d68cb4e2e8f3e08db11f469ea"
format_stamp: "Formatted at 2026-08-12 02:37:07 on dist-test-slave-rmhg"
server_key: "3ea73dc14c43acb436962727e40b0278"
server_key_iv: "fcfb56ad7c61c4288b8565b1e4720bb2"
server_key_version: "encryptionkey@0"
I20260812 02:37:07.032043 4201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "c0134a2d68cb4e2e8f3e08db11f469ea"
format_stamp: "Formatted at 2026-08-12 02:37:07 on dist-test-slave-rmhg"
server_key: "3ea73dc14c43acb436962727e40b0278"
server_key_iv: "fcfb56ad7c61c4288b8565b1e4720bb2"
server_key_version: "encryptionkey@0"
I20260812 02:37:07.036777 4201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260812 02:37:07.039930 4233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:07.041553 4201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:07.041690 4201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "c0134a2d68cb4e2e8f3e08db11f469ea"
format_stamp: "Formatted at 2026-08-12 02:37:07 on dist-test-slave-rmhg"
server_key: "3ea73dc14c43acb436962727e40b0278"
server_key_iv: "fcfb56ad7c61c4288b8565b1e4720bb2"
server_key_version: "encryptionkey@0"
I20260812 02:37:07.041814 4201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:07.059849 4201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:07.063716 4201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:07.064002 4201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:07.064929 4201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:07.066071 4201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:07.066126 4201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:07.066187 4201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:07.066215 4201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:07.089421 4201 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:33723
I20260812 02:37:07.089548 4418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:33723 every 8 connection(s)
I20260812 02:37:07.090534 4201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:37:07.098520 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 4201
I20260812 02:37:07.098639 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:37:07.098958 3582 external_mini_cluster.cc:1428] Setting key 148d17eb6669869e1cbc0d0dce212852
Aug 12 02:37:07 dist-test-slave-rmhg krb5kdc[3599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502227, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:07.107844 3738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:07.092861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:36999 (local address 127.3.127.190:33729)
0812 02:37:07.093055 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:07.093058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:07.094253 (+ 1195us) server_negotiation.cc:407] Connection header received
0812 02:37:07.096017 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:07.096021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:07.096081 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:07.096172 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:07.098012 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:07.098559 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:07.100572 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:07.100870 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:07.100970 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:07.104399 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:07.104402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:07.104404 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:07.104437 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:07.105836 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:07.106356 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:07.106358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:07.106360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:07.106432 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:07.106788 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:07.106790 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:07.106791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:07.106974 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:07.107048 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:07.107599 (+ 551us) server_negotiation.cc:299] Negotiation successful
0812 02:37:07.107713 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 02:37:07.108855 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:07.093273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:33729 (local address 127.3.127.131:36999)
0812 02:37:07.094042 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:07.094098 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:07.095722 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:07.096392 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:07.096401 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:07.096923 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:07.097788 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:07.097809 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:07.098730 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:07.098734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:07.100381 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:07.100396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:07.101049 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:07.101054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:07.101328 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:07.102076 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:07.102096 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:07.104182 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:07.105965 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:07.105972 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:07.105988 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:07.106271 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:07.106539 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:07.106542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:07.106543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:07.106708 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:07.107053 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:07.107058 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:07.107347 (+ 289us) client_negotiation.cc:769] Sending connection context
0812 02:37:07.107556 (+ 209us) client_negotiation.cc:240] Negotiation successful
0812 02:37:07.107868 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":224,"threads_started":1}
I20260812 02:37:07.110339 4419 heartbeater.cc:347] Connected to a master server at 127.3.127.190:33729
I20260812 02:37:07.110610 4419 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:07.111214 4419 heartbeater.cc:511] Master 127.3.127.190:33729 requested a full tablet report, sending...
I20260812 02:37:07.112351 3669 ts_manager.cc:194] Registered new tserver with Master: c0134a2d68cb4e2e8f3e08db11f469ea (127.3.127.131:33723)
I20260812 02:37:07.113011 3669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:36999
I20260812 02:37:07.114553 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 02:37:07 dist-test-slave-rmhg krb5kdc[3599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502227, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 02:37:07 dist-test-slave-rmhg krb5kdc[3599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502227, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.127.129@KRBTEST.COM
I20260812 02:37:07.145370 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:07.135296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59172 (local address 127.3.127.129:38495)
0812 02:37:07.135739 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:07.135745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:07.135773 (+ 28us) server_negotiation.cc:407] Connection header received
0812 02:37:07.135886 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:07.135894 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:07.136091 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:07.136239 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:07.137013 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:07.137843 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:07.138721 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:07.139127 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:07.139221 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:07.141670 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:07.141687 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:07.141706 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:07.141740 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:07.143360 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:07.143822 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:07.143827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:07.143828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:07.143916 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:07.144161 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:07.144163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:07.144164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:07.144454 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:07.144630 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:07.144953 (+ 323us) server_negotiation.cc:299] Negotiation successful
0812 02:37:07.145206 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":205,"threads_started":1}
W20260812 02:37:07.146596 3846 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59172
Aug 12 02:37:07 dist-test-slave-rmhg krb5kdc[3599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 02:37:07 dist-test-slave-rmhg krb5kdc[3599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.129@KRBTEST.COM
I20260812 02:37:07.176888 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:07.168224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59188 (local address 127.3.127.129:38495)
0812 02:37:07.168386 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:07.168391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:07.168505 (+ 114us) server_negotiation.cc:407] Connection header received
0812 02:37:07.168693 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:07.168696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:07.168744 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:07.168814 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:07.169609 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:07.170181 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:07.171020 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:07.171300 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:07.171397 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:07.173934 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:07.173938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:07.173940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:07.173971 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:07.175192 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:07.175650 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:07.175654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:07.175655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:07.175747 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:07.176014 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:07.176017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:07.176018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:07.176214 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:07.176340 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:07.176586 (+ 246us) server_negotiation.cc:299] Negotiation successful
0812 02:37:07.176728 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 02:37:07.178157 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 3775
I20260812 02:37:07.191915 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 3993
I20260812 02:37:07.229966 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 4201
I20260812 02:37:07.246676 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 3615
2026-08-12T02:37:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2818 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:37:07 dist-test-slave-rmhg krb5kdc[4444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:37:07 dist-test-slave-rmhg krb5kdc[4444](info): set up 2 sockets
Aug 12 02:37:07 dist-test-slave-rmhg krb5kdc[4444](info): commencing operation
krb5kdc: starting...
W20260812 02:37:08.428697 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:37:08 dist-test-slave-rmhg krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:37:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:37:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:37:08.560881 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:33793
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:33793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:08.680711 4471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:08.681032 4471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:08.681093 4471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:08.685756 4471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:08.685829 4471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:08.685858 4471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:08.685878 4471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:08.685907 4471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:37:08.691536 4471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:33793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:33793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:08.692708 4471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:08.693841 4471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:08.702831 4480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:08.702838 4483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:08.702939 4471 server_base.cc:1034] running on GCE node
W20260812 02:37:08.702831 4481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:08.703848 4471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:08.705236 4471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:08.706449 4471 hybrid_clock.cc:648] HybridClock initialized: now 1786502228706416 us; error 49 us; skew 500 ppm
Aug 12 02:37:08 dist-test-slave-rmhg krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502228, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:08.709570 4471 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:37:08.710650 4471 webserver.cc:460] Webserver started at http://127.3.127.190:45627/ using document root <none> and password file <none>
I20260812 02:37:08.711252 4471 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:08.711318 4471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:08.711520 4471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:08.713789 4471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "e5133ff9ec00447ebf6d7642bddcda20"
format_stamp: "Formatted at 2026-08-12 02:37:08 on dist-test-slave-rmhg"
server_key: "289df644452185e05f71137638360432"
server_key_iv: "a5319b8e6a75d54f0b365321d5176858"
server_key_version: "encryptionkey@0"
I20260812 02:37:08.714380 4471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "e5133ff9ec00447ebf6d7642bddcda20"
format_stamp: "Formatted at 2026-08-12 02:37:08 on dist-test-slave-rmhg"
server_key: "289df644452185e05f71137638360432"
server_key_iv: "a5319b8e6a75d54f0b365321d5176858"
server_key_version: "encryptionkey@0"
I20260812 02:37:08.718585 4471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:37:08.721266 4493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:08.722690 4471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:08.722833 4471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "e5133ff9ec00447ebf6d7642bddcda20"
format_stamp: "Formatted at 2026-08-12 02:37:08 on dist-test-slave-rmhg"
server_key: "289df644452185e05f71137638360432"
server_key_iv: "a5319b8e6a75d54f0b365321d5176858"
server_key_version: "encryptionkey@0"
I20260812 02:37:08.722956 4471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:08.736789 4471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:08.740288 4471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:08.740477 4471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:08.748168 4471 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:33793
I20260812 02:37:08.748202 4577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:33793 every 8 connection(s)
I20260812 02:37:08.749361 4471 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:37:08.752347 4579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:08.758520 4579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20: Bootstrap starting.
I20260812 02:37:08.758951 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 4471
I20260812 02:37:08.759063 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:37:08.759339 3582 external_mini_cluster.cc:1428] Setting key 02b7dc6e6f0bafca755b395c121c2e18
I20260812 02:37:08.761641 4579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:08.762542 4579 log.cc:824] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:08.764858 4579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20: No bootstrap required, opened a new log
I20260812 02:37:08.769006 4579 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5133ff9ec00447ebf6d7642bddcda20" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33793 } }
I20260812 02:37:08.769328 4579 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:08.769407 4579 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5133ff9ec00447ebf6d7642bddcda20, State: Initialized, Role: FOLLOWER
Aug 12 02:37:08 dist-test-slave-rmhg krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:08.769905 4579 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5133ff9ec00447ebf6d7642bddcda20" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33793 } }
I20260812 02:37:08.770047 4579 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:37:08.770133 4579 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:37:08.770241 4579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:08.771490 4579 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5133ff9ec00447ebf6d7642bddcda20" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33793 } }
I20260812 02:37:08.771898 4579 leader_election.cc:302] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5133ff9ec00447ebf6d7642bddcda20; no voters:
I20260812 02:37:08.772341 4579 leader_election.cc:288] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:37:08.772454 4585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:08.772681 4585 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20 [term 1 LEADER]: Becoming Leader. State: Replica: e5133ff9ec00447ebf6d7642bddcda20, State: Running, Role: LEADER
I20260812 02:37:08.773026 4585 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5133ff9ec00447ebf6d7642bddcda20" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33793 } }
I20260812 02:37:08.773835 4579 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:37:08.775686 4586 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5133ff9ec00447ebf6d7642bddcda20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5133ff9ec00447ebf6d7642bddcda20" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33793 } } }
I20260812 02:37:08.775847 4586 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:08.776147 4587 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5133ff9ec00447ebf6d7642bddcda20. Latest consensus state: current_term: 1 leader_uuid: "e5133ff9ec00447ebf6d7642bddcda20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5133ff9ec00447ebf6d7642bddcda20" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33793 } } }
I20260812 02:37:08.776242 4587 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:08.776510 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:08.760986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36008 (local address 127.3.127.190:33793)
0812 02:37:08.761575 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:08.761592 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:08.761635 (+ 43us) server_negotiation.cc:407] Connection header received
0812 02:37:08.762449 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:08.762472 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:08.762904 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:08.763280 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:08.764301 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:08.765493 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:08.766718 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:08.767396 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:08.767590 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:08.770210 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:08.770218 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:08.770237 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:08.770279 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:08.772365 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:08.772928 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:08.772934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:08.772941 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:08.773087 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:08.773424 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:08.773427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:08.773429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:08.774016 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:08.774224 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:08.774759 (+ 535us) server_negotiation.cc:299] Negotiation successful
0812 02:37:08.775023 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":192,"threads_started":1}
I20260812 02:37:08.776525 4598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:37:08.780056 4598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:37:08.786494 4598 catalog_manager.cc:1347] Generated new cluster ID: b8d8c7086406438ba14d673ef0e06bed
I20260812 02:37:08.786581 4598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:37:08.805651 4598 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:37:08.806849 4598 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:37:08.812544 4598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5133ff9ec00447ebf6d7642bddcda20: Generated new TSK 0
I20260812 02:37:08.813350 4598 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
I20260812 02:37:08.845263 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33793
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:08.968289 4616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:08.968600 4616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:08.968654 4616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:08.972891 4616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:08.972968 4616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:08.973033 4616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:37:08.978830 4616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33793
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:08.980050 4616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:08.981140 4616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:08.991444 4624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:08.991444 4625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:08.991459 4629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:08.991701 4616 server_base.cc:1034] running on GCE node
I20260812 02:37:08.992172 4616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:08.992887 4616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:08.994047 4616 hybrid_clock.cc:648] HybridClock initialized: now 1786502228994015 us; error 44 us; skew 500 ppm
Aug 12 02:37:08 dist-test-slave-rmhg krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502228, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:08.997009 4616 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:37:08.998073 4616 webserver.cc:460] Webserver started at http://127.3.127.129:39033/ using document root <none> and password file <none>
I20260812 02:37:08.998677 4616 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:08.998749 4616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:08.998939 4616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:09.001482 4616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "58c3829c75374caba5d0765d913e5e03"
format_stamp: "Formatted at 2026-08-12 02:37:08 on dist-test-slave-rmhg"
server_key: "45068b6146277b45f1f6c8cf31778149"
server_key_iv: "f86cdb03a6514aa71b0e1cf7216d0c6d"
server_key_version: "encryptionkey@0"
I20260812 02:37:09.002074 4616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "58c3829c75374caba5d0765d913e5e03"
format_stamp: "Formatted at 2026-08-12 02:37:08 on dist-test-slave-rmhg"
server_key: "45068b6146277b45f1f6c8cf31778149"
server_key_iv: "f86cdb03a6514aa71b0e1cf7216d0c6d"
server_key_version: "encryptionkey@0"
I20260812 02:37:09.006237 4616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:37:09.008899 4639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:09.010277 4616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 02:37:09.010397 4616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "58c3829c75374caba5d0765d913e5e03"
format_stamp: "Formatted at 2026-08-12 02:37:08 on dist-test-slave-rmhg"
server_key: "45068b6146277b45f1f6c8cf31778149"
server_key_iv: "f86cdb03a6514aa71b0e1cf7216d0c6d"
server_key_version: "encryptionkey@0"
I20260812 02:37:09.010521 4616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:09.041124 4616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:09.044654 4616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:09.044852 4616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:09.045567 4616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:09.046495 4616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:09.046561 4616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:09.046636 4616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:09.046661 4616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:09.055552 4616 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:36221
I20260812 02:37:09.055663 4832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:36221 every 8 connection(s)
I20260812 02:37:09.056525 4616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:37:09.064414 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 4616
I20260812 02:37:09.064560 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:37:09.064833 3582 external_mini_cluster.cc:1428] Setting key 6f2ca14b6c0d516fdbdce2e51b5dab63
Aug 12 02:37:09 dist-test-slave-rmhg krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502228, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:09.071947 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:09.058629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:36359 (local address 127.3.127.190:33793)
0812 02:37:09.058773 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:09.058777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:09.059732 (+ 955us) server_negotiation.cc:407] Connection header received
0812 02:37:09.061138 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:09.061142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:09.061227 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:09.061304 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:09.062641 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.063193 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.064536 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.064837 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.064967 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:09.067865 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:09.067868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:09.067870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:09.067898 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:09.069615 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:09.070206 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:09.070210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:09.070212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:09.070323 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:09.070654 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:09.070657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:09.070658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:09.070908 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:09.071022 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:09.071591 (+ 569us) server_negotiation.cc:299] Negotiation successful
0812 02:37:09.071753 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 02:37:09.072901 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:09.059004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:33793 (local address 127.3.127.129:36359)
0812 02:37:09.059536 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:09.059585 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:09.060908 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:09.061501 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:09.061509 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:09.061916 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:09.062476 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:09.062496 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.063323 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.063326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:09.064357 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:09.064371 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.065049 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.065054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:09.065301 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:09.066003 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:09.066022 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:09.067672 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:09.069756 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:09.069762 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:09.069779 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:09.070079 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:09.070415 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:09.070417 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:09.070418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:09.070568 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:09.071027 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:09.071034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:09.071321 (+ 287us) client_negotiation.cc:769] Sending connection context
0812 02:37:09.071491 (+ 170us) client_negotiation.cc:240] Negotiation successful
0812 02:37:09.071867 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":136,"threads_started":1}
I20260812 02:37:09.074523 4835 heartbeater.cc:347] Connected to a master server at 127.3.127.190:33793
I20260812 02:37:09.074798 4835 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:09.075441 4835 heartbeater.cc:511] Master 127.3.127.190:33793 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.127.130@KRBTEST.COM; defaulting to no policy
I20260812 02:37:09.077301 4524 ts_manager.cc:194] Registered new tserver with Master: 58c3829c75374caba5d0765d913e5e03 (127.3.127.129:36221)
I20260812 02:37:09.078574 4524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:36359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
I20260812 02:37:09.097334 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33793
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:09.218897 4848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:09.219228 4848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:09.219275 4848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:09.223744 4848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:09.223841 4848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:09.223905 4848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:37:09.229923 4848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33793
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:09.231168 4848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:09.232388 4848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:09.243321 4856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:09.243265 4858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:09.243268 4854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:09.243569 4848 server_base.cc:1034] running on GCE node
I20260812 02:37:09.244127 4848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:09.244968 4848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:09.246150 4848 hybrid_clock.cc:648] HybridClock initialized: now 1786502229246121 us; error 49 us; skew 500 ppm
Aug 12 02:37:09 dist-test-slave-rmhg krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502229, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:09.249545 4848 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:37:09.250625 4848 webserver.cc:460] Webserver started at http://127.3.127.130:35969/ using document root <none> and password file <none>
I20260812 02:37:09.251240 4848 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:09.251318 4848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:09.251557 4848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:09.253904 4848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "114af09c02fe4a928166df10dcffa4d0"
format_stamp: "Formatted at 2026-08-12 02:37:09 on dist-test-slave-rmhg"
server_key: "3ce86ba7d4d97cbb55e889f4ae36aa8f"
server_key_iv: "6a660a9b92e41e90220a9d683ea63e5e"
server_key_version: "encryptionkey@0"
I20260812 02:37:09.254513 4848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "114af09c02fe4a928166df10dcffa4d0"
format_stamp: "Formatted at 2026-08-12 02:37:09 on dist-test-slave-rmhg"
server_key: "3ce86ba7d4d97cbb55e889f4ae36aa8f"
server_key_iv: "6a660a9b92e41e90220a9d683ea63e5e"
server_key_version: "encryptionkey@0"
I20260812 02:37:09.258860 4848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:37:09.261693 4871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:09.263017 4848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:09.263172 4848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "114af09c02fe4a928166df10dcffa4d0"
format_stamp: "Formatted at 2026-08-12 02:37:09 on dist-test-slave-rmhg"
server_key: "3ce86ba7d4d97cbb55e889f4ae36aa8f"
server_key_iv: "6a660a9b92e41e90220a9d683ea63e5e"
server_key_version: "encryptionkey@0"
I20260812 02:37:09.263286 4848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:09.280786 4848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:09.284047 4848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:09.284255 4848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:09.284972 4848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:09.285939 4848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:09.285990 4848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:09.286031 4848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:09.286046 4848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:09.295113 5063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:46099 every 8 connection(s)
I20260812 02:37:09.295161 4848 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:46099
I20260812 02:37:09.296340 4848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:37:09.305860 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 4848
I20260812 02:37:09.305999 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:37:09.306326 3582 external_mini_cluster.cc:1428] Setting key 16c2418dfef356917fc2a3de841c80a5
Aug 12 02:37:09 dist-test-slave-rmhg krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502229, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:09.312633 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:09.298758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:48097 (local address 127.3.127.190:33793)
0812 02:37:09.298916 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:09.298920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:09.299931 (+ 1011us) server_negotiation.cc:407] Connection header received
0812 02:37:09.301406 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:09.301409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:09.301463 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:09.301542 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:09.302876 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.303437 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.304861 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.305135 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.305283 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:09.308362 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:09.308366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:09.308369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:09.308403 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:09.310064 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:09.310821 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:09.310824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:09.310826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:09.310920 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:09.311369 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:09.311373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:09.311374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:09.311600 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:09.311684 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:09.312321 (+ 637us) server_negotiation.cc:299] Negotiation successful
0812 02:37:09.312475 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 02:37:09.313599 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:09.299079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:33793 (local address 127.3.127.130:48097)
0812 02:37:09.299693 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:09.299766 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:09.301127 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:09.301724 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:09.301732 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:09.302142 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:09.302699 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:09.302716 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.303665 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.303668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:09.304691 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:09.304706 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.305364 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.305368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:09.305633 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:09.306386 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:09.306411 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:09.308164 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:09.310231 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:09.310240 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:09.310264 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:09.310691 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:09.311047 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:09.311050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:09.311052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:09.311266 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:09.311710 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:09.311717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:09.312049 (+ 332us) client_negotiation.cc:769] Sending connection context
0812 02:37:09.312239 (+ 190us) client_negotiation.cc:240] Negotiation successful
0812 02:37:09.312596 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":139,"threads_started":1}
I20260812 02:37:09.314916 5067 heartbeater.cc:347] Connected to a master server at 127.3.127.190:33793
I20260812 02:37:09.315248 5067 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:09.315972 5067 heartbeater.cc:511] Master 127.3.127.190:33793 requested a full tablet report, sending...
I20260812 02:37:09.317386 4524 ts_manager.cc:194] Registered new tserver with Master: 114af09c02fe4a928166df10dcffa4d0 (127.3.127.130:46099)
I20260812 02:37:09.318007 4524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:48097
WARNING: no policy specified for kudu/127.3.127.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
I20260812 02:37:09.340795 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33793
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:09.462880 5076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:09.463200 5076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:09.463354 5076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:09.467861 5076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:09.467941 5076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:09.468020 5076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:37:09.473735 5076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33793
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:09.474988 5076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:09.476109 5076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:09.486398 5082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:09.486403 5087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:09.486403 5081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:09.486903 5076 server_base.cc:1034] running on GCE node
I20260812 02:37:09.487365 5076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:09.488078 5076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:09.489251 5076 hybrid_clock.cc:648] HybridClock initialized: now 1786502229489222 us; error 45 us; skew 500 ppm
Aug 12 02:37:09 dist-test-slave-rmhg krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502229, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:09.492154 5076 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:37:09.493212 5076 webserver.cc:460] Webserver started at http://127.3.127.131:35619/ using document root <none> and password file <none>
I20260812 02:37:09.493826 5076 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:09.493899 5076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:09.494119 5076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:09.496358 5076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "aef5ea37f35a439b9de2939f0426a6d6"
format_stamp: "Formatted at 2026-08-12 02:37:09 on dist-test-slave-rmhg"
server_key: "e6de7ce526b66fba4a29ebda39749dd0"
server_key_iv: "33bc61489b7b1f80dac099fe4e43e8c2"
server_key_version: "encryptionkey@0"
I20260812 02:37:09.496949 5076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "aef5ea37f35a439b9de2939f0426a6d6"
format_stamp: "Formatted at 2026-08-12 02:37:09 on dist-test-slave-rmhg"
server_key: "e6de7ce526b66fba4a29ebda39749dd0"
server_key_iv: "33bc61489b7b1f80dac099fe4e43e8c2"
server_key_version: "encryptionkey@0"
I20260812 02:37:09.501106 5076 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:37:09.503754 5097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:09.505129 5076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:37:09.505321 5076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "aef5ea37f35a439b9de2939f0426a6d6"
format_stamp: "Formatted at 2026-08-12 02:37:09 on dist-test-slave-rmhg"
server_key: "e6de7ce526b66fba4a29ebda39749dd0"
server_key_iv: "33bc61489b7b1f80dac099fe4e43e8c2"
server_key_version: "encryptionkey@0"
I20260812 02:37:09.505437 5076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:09.529654 5076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:09.532915 5076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:09.533120 5076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:09.533795 5076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:09.534684 5076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:09.534737 5076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:09.534796 5076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:09.534824 5076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:09.544097 5076 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:43997
I20260812 02:37:09.544118 5273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:43997 every 8 connection(s)
I20260812 02:37:09.545192 5076 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:37:09.549321 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 5076
I20260812 02:37:09.549453 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:37:09.549777 3582 external_mini_cluster.cc:1428] Setting key ccf456cf0c9c45906003c1f0135eb7fa
Aug 12 02:37:09 dist-test-slave-rmhg krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502229, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:09.560503 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:09.547273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:39757 (local address 127.3.127.190:33793)
0812 02:37:09.547399 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:09.547403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:09.548416 (+ 1013us) server_negotiation.cc:407] Connection header received
0812 02:37:09.549975 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:09.549979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:09.550043 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:09.550129 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:09.551538 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.552259 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.553714 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.553958 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.554056 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:09.556926 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:09.556930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:09.556932 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:09.556963 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:09.558350 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:09.558951 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:09.558954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:09.558955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:09.559042 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:09.559420 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:09.559422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:09.559423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:09.559612 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:09.559693 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:09.560261 (+ 568us) server_negotiation.cc:299] Negotiation successful
0812 02:37:09.560374 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 02:37:09.561551 5277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:09.547742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:33793 (local address 127.3.127.131:39757)
0812 02:37:09.548253 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:09.548300 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:09.549693 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:09.550332 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:09.550339 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:09.550791 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:09.551385 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:09.551401 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.552396 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.552398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:09.553544 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:09.553557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.554164 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.554168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:09.554347 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:09.555022 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:09.555041 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:09.556722 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:09.558525 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:09.558531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:09.558546 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:09.558850 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:09.559158 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:09.559161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:09.559163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:09.559337 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:09.559740 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:09.559746 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:09.560033 (+ 287us) client_negotiation.cc:769] Sending connection context
0812 02:37:09.560226 (+ 193us) client_negotiation.cc:240] Negotiation successful
0812 02:37:09.560531 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":135,"threads_started":1}
I20260812 02:37:09.562978 5274 heartbeater.cc:347] Connected to a master server at 127.3.127.190:33793
I20260812 02:37:09.563272 5274 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:09.563934 5274 heartbeater.cc:511] Master 127.3.127.190:33793 requested a full tablet report, sending...
I20260812 02:37:09.565032 4524 ts_manager.cc:194] Registered new tserver with Master: aef5ea37f35a439b9de2939f0426a6d6 (127.3.127.131:43997)
I20260812 02:37:09.565724 4524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:39757
I20260812 02:37:09.576653 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:37:09.590318 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:09.581682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.3.127.190:33793)
0812 02:37:09.581867 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:09.581870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:09.582165 (+ 295us) server_negotiation.cc:407] Connection header received
0812 02:37:09.582311 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:09.582315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:09.582368 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:09.582438 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:09.583252 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.583797 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.585113 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.585412 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.585522 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:09.587045 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:09.587049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:09.587051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:09.587081 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:09.588463 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:09.588954 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:09.588957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:09.588958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:09.589054 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:09.589350 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:09.589353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:09.589354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:09.589649 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:09.589761 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:09.590059 (+ 298us) server_negotiation.cc:299] Negotiation successful
0812 02:37:09.590192 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 02:37:09.596797 4524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:37:09.600698 4524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:37:09.618754 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:09.612750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44144 (local address 127.3.127.129:36221)
0812 02:37:09.613768 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:09.613777 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:09.613802 (+ 25us) server_negotiation.cc:407] Connection header received
0812 02:37:09.613887 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:09.613894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:09.614162 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:09.614349 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:09.614674 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.615968 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.617327 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.618012 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.618129 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:09.618344 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:09.618472 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 02:37:09.618581 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":901,"thread_start_us":130,"threads_started":1}
I20260812 02:37:09.618747 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:09.612525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:36221 (local address 127.0.0.1:44144)
0812 02:37:09.613332 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:09.613346 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:09.613440 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:09.614338 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:09.614344 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:09.614371 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:09.614542 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:09.614549 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.616101 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.616104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:09.617212 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:09.617225 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.618149 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.618153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:09.618279 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:09.618322 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 02:37:09.618437 (+ 115us) client_negotiation.cc:240] Negotiation successful
0812 02:37:09.618589 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"spinlock_wait_cycles":768,"thread_start_us":137,"threads_started":1}
I20260812 02:37:09.619082 5295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:09.613104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:43997 (local address 127.0.0.1:42394)
0812 02:37:09.613744 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:09.613759 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:09.613879 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:09.614740 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:09.614742 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:09.614762 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:09.614888 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:09.614896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.616283 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.616286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:09.617589 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:09.617601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.618697 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.618701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:09.618801 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:09.618828 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 02:37:09.618878 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 02:37:09.618949 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":3,"thread_start_us":170,"threads_started":1}
I20260812 02:37:09.619529 5290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:09.612240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:46099 (local address 127.0.0.1:39836)
0812 02:37:09.613241 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:09.613302 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:09.613441 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:09.614390 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:09.614393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:09.614412 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:09.614586 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:09.614594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.616543 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.616547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:09.617756 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:09.617764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.619164 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.619168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:09.619256 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:09.619280 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 02:37:09.619329 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 02:37:09.619408 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":20,"thread_start_us":112,"threads_started":1}
I20260812 02:37:09.619936 5297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:09.613668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42394 (local address 127.3.127.131:43997)
0812 02:37:09.614250 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:09.614275 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:09.614300 (+ 25us) server_negotiation.cc:407] Connection header received
0812 02:37:09.614372 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:09.614377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:09.614562 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:09.614759 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:09.614976 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.616108 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.617725 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.618556 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.619184 (+ 628us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:09.619323 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:09.619365 (+ 42us) server_negotiation.cc:299] Negotiation successful
0812 02:37:09.619781 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":296,"threads_started":1}
I20260812 02:37:09.620527 5291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:09.612438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39836 (local address 127.3.127.130:46099)
0812 02:37:09.613812 (+ 1374us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:09.613858 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:09.613880 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:37:09.613948 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:09.613952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:09.614193 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:09.614748 (+ 555us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:09.615009 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.616352 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.617868 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.619033 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.619948 (+ 915us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:09.620103 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:09.620151 (+ 48us) server_negotiation.cc:299] Negotiation successful
0812 02:37:09.620236 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1277,"thread_start_us":141,"threads_started":1}
I20260812 02:37:09.622944 4722 tablet_service.cc:1467] Processing CreateTablet for tablet 3446f4e45eae4aefb8e8d8b5408481ed (DEFAULT_TABLE table=test-table [id=4af918314dd34af4bcb6249de72c2ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:09.623124 5177 tablet_service.cc:1467] Processing CreateTablet for tablet 3446f4e45eae4aefb8e8d8b5408481ed (DEFAULT_TABLE table=test-table [id=4af918314dd34af4bcb6249de72c2ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:09.623346 4960 tablet_service.cc:1467] Processing CreateTablet for tablet 3446f4e45eae4aefb8e8d8b5408481ed (DEFAULT_TABLE table=test-table [id=4af918314dd34af4bcb6249de72c2ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:09.624363 4722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3446f4e45eae4aefb8e8d8b5408481ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:09.624380 4960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3446f4e45eae4aefb8e8d8b5408481ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:09.624383 5177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3446f4e45eae4aefb8e8d8b5408481ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:09.632956 5298 tablet_bootstrap.cc:492] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03: Bootstrap starting.
I20260812 02:37:09.635133 5299 tablet_bootstrap.cc:492] T 3446f4e45eae4aefb8e8d8b5408481ed P aef5ea37f35a439b9de2939f0426a6d6: Bootstrap starting.
I20260812 02:37:09.635133 5300 tablet_bootstrap.cc:492] T 3446f4e45eae4aefb8e8d8b5408481ed P 114af09c02fe4a928166df10dcffa4d0: Bootstrap starting.
I20260812 02:37:09.636368 5298 tablet_bootstrap.cc:654] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:09.637799 5298 log.cc:824] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:09.639084 5299 tablet_bootstrap.cc:654] T 3446f4e45eae4aefb8e8d8b5408481ed P aef5ea37f35a439b9de2939f0426a6d6: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:09.640065 5298 tablet_bootstrap.cc:492] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03: No bootstrap required, opened a new log
I20260812 02:37:09.640275 5298 ts_tablet_manager.cc:1397] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260812 02:37:09.640340 5299 log.cc:824] T 3446f4e45eae4aefb8e8d8b5408481ed P aef5ea37f35a439b9de2939f0426a6d6: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:09.640923 5300 tablet_bootstrap.cc:654] T 3446f4e45eae4aefb8e8d8b5408481ed P 114af09c02fe4a928166df10dcffa4d0: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:09.642710 5300 log.cc:824] T 3446f4e45eae4aefb8e8d8b5408481ed P 114af09c02fe4a928166df10dcffa4d0: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:09.642933 5299 tablet_bootstrap.cc:492] T 3446f4e45eae4aefb8e8d8b5408481ed P aef5ea37f35a439b9de2939f0426a6d6: No bootstrap required, opened a new log
I20260812 02:37:09.643214 5299 ts_tablet_manager.cc:1397] T 3446f4e45eae4aefb8e8d8b5408481ed P aef5ea37f35a439b9de2939f0426a6d6: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 02:37:09.645574 5300 tablet_bootstrap.cc:492] T 3446f4e45eae4aefb8e8d8b5408481ed P 114af09c02fe4a928166df10dcffa4d0: No bootstrap required, opened a new log
I20260812 02:37:09.645813 5300 ts_tablet_manager.cc:1397] T 3446f4e45eae4aefb8e8d8b5408481ed P 114af09c02fe4a928166df10dcffa4d0: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260812 02:37:09.646965 5298 raft_consensus.cc:348] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114af09c02fe4a928166df10dcffa4d0" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 46099 } } peers { permanent_uuid: "aef5ea37f35a439b9de2939f0426a6d6" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 43997 } } peers { permanent_uuid: "58c3829c75374caba5d0765d913e5e03" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 36221 } }
I20260812 02:37:09.647225 5298 raft_consensus.cc:374] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:09.647290 5298 raft_consensus.cc:729] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58c3829c75374caba5d0765d913e5e03, State: Initialized, Role: FOLLOWER
I20260812 02:37:09.647746 5298 consensus_queue.cc:260] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114af09c02fe4a928166df10dcffa4d0" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 46099 } } peers { permanent_uuid: "aef5ea37f35a439b9de2939f0426a6d6" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 43997 } } peers { permanent_uuid: "58c3829c75374caba5d0765d913e5e03" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 36221 } }
I20260812 02:37:09.648650 4835 heartbeater.cc:503] Master 127.3.127.190:33793 was elected leader, sending a full tablet report...
I20260812 02:37:09.648943 5298 ts_tablet_manager.cc:1428] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260812 02:37:09.649621 5299 raft_consensus.cc:348] T 3446f4e45eae4aefb8e8d8b5408481ed P aef5ea37f35a439b9de2939f0426a6d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114af09c02fe4a928166df10dcffa4d0" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 46099 } } peers { permanent_uuid: "aef5ea37f35a439b9de2939f0426a6d6" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 43997 } } peers { permanent_uuid: "58c3829c75374caba5d0765d913e5e03" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 36221 } }
I20260812 02:37:09.649989 5299 raft_consensus.cc:374] T 3446f4e45eae4aefb8e8d8b5408481ed P aef5ea37f35a439b9de2939f0426a6d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:09.650055 5299 raft_consensus.cc:729] T 3446f4e45eae4aefb8e8d8b5408481ed P aef5ea37f35a439b9de2939f0426a6d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aef5ea37f35a439b9de2939f0426a6d6, State: Initialized, Role: FOLLOWER
I20260812 02:37:09.650636 5299 consensus_queue.cc:260] T 3446f4e45eae4aefb8e8d8b5408481ed P aef5ea37f35a439b9de2939f0426a6d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114af09c02fe4a928166df10dcffa4d0" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 46099 } } peers { permanent_uuid: "aef5ea37f35a439b9de2939f0426a6d6" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 43997 } } peers { permanent_uuid: "58c3829c75374caba5d0765d913e5e03" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 36221 } }
I20260812 02:37:09.651515 5274 heartbeater.cc:503] Master 127.3.127.190:33793 was elected leader, sending a full tablet report...
I20260812 02:37:09.651880 5299 ts_tablet_manager.cc:1428] T 3446f4e45eae4aefb8e8d8b5408481ed P aef5ea37f35a439b9de2939f0426a6d6: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260812 02:37:09.652267 5300 raft_consensus.cc:348] T 3446f4e45eae4aefb8e8d8b5408481ed P 114af09c02fe4a928166df10dcffa4d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114af09c02fe4a928166df10dcffa4d0" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 46099 } } peers { permanent_uuid: "aef5ea37f35a439b9de2939f0426a6d6" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 43997 } } peers { permanent_uuid: "58c3829c75374caba5d0765d913e5e03" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 36221 } }
I20260812 02:37:09.652608 5300 raft_consensus.cc:374] T 3446f4e45eae4aefb8e8d8b5408481ed P 114af09c02fe4a928166df10dcffa4d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:09.652674 5300 raft_consensus.cc:729] T 3446f4e45eae4aefb8e8d8b5408481ed P 114af09c02fe4a928166df10dcffa4d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 114af09c02fe4a928166df10dcffa4d0, State: Initialized, Role: FOLLOWER
I20260812 02:37:09.653118 5300 consensus_queue.cc:260] T 3446f4e45eae4aefb8e8d8b5408481ed P 114af09c02fe4a928166df10dcffa4d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114af09c02fe4a928166df10dcffa4d0" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 46099 } } peers { permanent_uuid: "aef5ea37f35a439b9de2939f0426a6d6" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 43997 } } peers { permanent_uuid: "58c3829c75374caba5d0765d913e5e03" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 36221 } }
I20260812 02:37:09.654161 5067 heartbeater.cc:503] Master 127.3.127.190:33793 was elected leader, sending a full tablet report...
I20260812 02:37:09.654249 5300 ts_tablet_manager.cc:1428] T 3446f4e45eae4aefb8e8d8b5408481ed P 114af09c02fe4a928166df10dcffa4d0: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260812 02:37:09.785373 5307 raft_consensus.cc:482] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:37:09.785679 5307 raft_consensus.cc:504] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114af09c02fe4a928166df10dcffa4d0" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 46099 } } peers { permanent_uuid: "aef5ea37f35a439b9de2939f0426a6d6" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 43997 } } peers { permanent_uuid: "58c3829c75374caba5d0765d913e5e03" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 36221 } }
I20260812 02:37:09.787132 5307 leader_election.cc:288] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 114af09c02fe4a928166df10dcffa4d0 (127.3.127.130:46099), aef5ea37f35a439b9de2939f0426a6d6 (127.3.127.131:43997)
I20260812 02:37:09.792021 5291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:09.787604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:40043 (local address 127.3.127.130:46099)
0812 02:37:09.787761 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:09.787765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:09.788147 (+ 382us) server_negotiation.cc:407] Connection header received
0812 02:37:09.788429 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:09.788433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:09.788498 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:09.788607 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:09.789055 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.789630 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.791002 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.791411 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.791521 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:09.791579 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:09.791827 (+ 248us) server_negotiation.cc:299] Negotiation successful
0812 02:37:09.791908 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 02:37:09.792097 5314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:09.787507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:46099 (local address 127.3.127.129:40043)
0812 02:37:09.788001 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:09.788016 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:09.788251 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:09.788638 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:09.788641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:09.788665 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:09.788930 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:09.788938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.789743 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.789746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:09.790877 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:09.790889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.791529 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.791532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:09.791691 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:09.791723 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 02:37:09.791798 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 02:37:09.791871 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":211,"threads_started":1}
I20260812 02:37:09.792531 5297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:09.788188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:49725 (local address 127.3.127.131:43997)
0812 02:37:09.788311 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:09.788316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:09.788793 (+ 477us) server_negotiation.cc:407] Connection header received
0812 02:37:09.788900 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:09.788903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:09.788947 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:09.789008 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:09.789473 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.789961 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.791350 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.791869 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.791979 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:09.792029 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:09.792321 (+ 292us) server_negotiation.cc:299] Negotiation successful
0812 02:37:09.792409 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 02:37:09.792979 5317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:09.788066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:43997 (local address 127.3.127.129:49725)
0812 02:37:09.788674 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:09.788691 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:09.788807 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:09.789033 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:09.789036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:09.789051 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:09.789348 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:09.789359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.790072 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.790075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:09.791217 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:09.791228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.791989 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.791992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:09.792168 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:09.792196 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 02:37:09.792702 (+ 506us) client_negotiation.cc:240] Negotiation successful
0812 02:37:09.792781 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":169,"threads_started":1}
I20260812 02:37:09.793004 4987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3446f4e45eae4aefb8e8d8b5408481ed" candidate_uuid: "58c3829c75374caba5d0765d913e5e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "114af09c02fe4a928166df10dcffa4d0" is_pre_election: true
I20260812 02:37:09.793623 4987 raft_consensus.cc:2454] T 3446f4e45eae4aefb8e8d8b5408481ed P 114af09c02fe4a928166df10dcffa4d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58c3829c75374caba5d0765d913e5e03 in term 0.
I20260812 02:37:09.793725 5205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3446f4e45eae4aefb8e8d8b5408481ed" candidate_uuid: "58c3829c75374caba5d0765d913e5e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aef5ea37f35a439b9de2939f0426a6d6" is_pre_election: true
I20260812 02:37:09.794041 5205 raft_consensus.cc:2454] T 3446f4e45eae4aefb8e8d8b5408481ed P aef5ea37f35a439b9de2939f0426a6d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58c3829c75374caba5d0765d913e5e03 in term 0.
I20260812 02:37:09.794412 4645 leader_election.cc:302] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 114af09c02fe4a928166df10dcffa4d0, 58c3829c75374caba5d0765d913e5e03; no voters:
I20260812 02:37:09.794734 5307 raft_consensus.cc:2781] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:37:09.794832 5307 raft_consensus.cc:482] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:37:09.794884 5307 raft_consensus.cc:3037] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03 [term 0 FOLLOWER]: Advancing to term 1
W20260812 02:37:09.796578 5275 tablet.cc:2369] T 3446f4e45eae4aefb8e8d8b5408481ed P aef5ea37f35a439b9de2939f0426a6d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:37:09.796659 5307 raft_consensus.cc:504] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114af09c02fe4a928166df10dcffa4d0" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 46099 } } peers { permanent_uuid: "aef5ea37f35a439b9de2939f0426a6d6" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 43997 } } peers { permanent_uuid: "58c3829c75374caba5d0765d913e5e03" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 36221 } }
I20260812 02:37:09.797086 5307 leader_election.cc:288] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03 [CANDIDATE]: Term 1 election: Requested vote from peers 114af09c02fe4a928166df10dcffa4d0 (127.3.127.130:46099), aef5ea37f35a439b9de2939f0426a6d6 (127.3.127.131:43997)
I20260812 02:37:09.797580 4987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3446f4e45eae4aefb8e8d8b5408481ed" candidate_uuid: "58c3829c75374caba5d0765d913e5e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "114af09c02fe4a928166df10dcffa4d0"
W20260812 02:37:09.797644 5068 tablet.cc:2369] T 3446f4e45eae4aefb8e8d8b5408481ed P 114af09c02fe4a928166df10dcffa4d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:37:09.797725 5205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3446f4e45eae4aefb8e8d8b5408481ed" candidate_uuid: "58c3829c75374caba5d0765d913e5e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aef5ea37f35a439b9de2939f0426a6d6"
I20260812 02:37:09.797758 4987 raft_consensus.cc:3037] T 3446f4e45eae4aefb8e8d8b5408481ed P 114af09c02fe4a928166df10dcffa4d0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:09.797855 5205 raft_consensus.cc:3037] T 3446f4e45eae4aefb8e8d8b5408481ed P aef5ea37f35a439b9de2939f0426a6d6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:09.799846 4987 raft_consensus.cc:2454] T 3446f4e45eae4aefb8e8d8b5408481ed P 114af09c02fe4a928166df10dcffa4d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58c3829c75374caba5d0765d913e5e03 in term 1.
I20260812 02:37:09.799881 5205 raft_consensus.cc:2454] T 3446f4e45eae4aefb8e8d8b5408481ed P aef5ea37f35a439b9de2939f0426a6d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58c3829c75374caba5d0765d913e5e03 in term 1.
I20260812 02:37:09.800201 4645 leader_election.cc:302] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58c3829c75374caba5d0765d913e5e03, aef5ea37f35a439b9de2939f0426a6d6; no voters:
I20260812 02:37:09.800424 5307 raft_consensus.cc:2781] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:09.800771 5307 raft_consensus.cc:686] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03 [term 1 LEADER]: Becoming Leader. State: Replica: 58c3829c75374caba5d0765d913e5e03, State: Running, Role: LEADER
I20260812 02:37:09.801226 5307 consensus_queue.cc:237] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114af09c02fe4a928166df10dcffa4d0" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 46099 } } peers { permanent_uuid: "aef5ea37f35a439b9de2939f0426a6d6" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 43997 } } peers { permanent_uuid: "58c3829c75374caba5d0765d913e5e03" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 36221 } }
I20260812 02:37:09.804960 4524 catalog_manager.cc:5495] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58c3829c75374caba5d0765d913e5e03 (127.3.127.129). New cstate: current_term: 1 leader_uuid: "58c3829c75374caba5d0765d913e5e03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "114af09c02fe4a928166df10dcffa4d0" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aef5ea37f35a439b9de2939f0426a6d6" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 43997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58c3829c75374caba5d0765d913e5e03" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 36221 } health_report { overall_health: HEALTHY } } }
W20260812 02:37:09.808079 4836 tablet.cc:2369] T 3446f4e45eae4aefb8e8d8b5408481ed P 58c3829c75374caba5d0765d913e5e03: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 02:37:09 dist-test-slave-rmhg krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.129@KRBTEST.COM
I20260812 02:37:09.859375 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:09.848649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46844 (local address 127.3.127.129:36221)
0812 02:37:09.848801 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:09.848805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:09.849201 (+ 396us) server_negotiation.cc:407] Connection header received
0812 02:37:09.849342 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:09.849344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:09.849403 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:09.849470 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:09.850383 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.850912 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.852037 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.852275 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.852367 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:09.855280 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:09.855299 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:09.855318 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:09.855351 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:09.857454 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:09.857946 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:09.857953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:09.857954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:09.858047 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:09.858330 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:09.858334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:09.858336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:09.858654 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:09.858889 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:09.859076 (+ 187us) server_negotiation.cc:299] Negotiation successful
0812 02:37:09.859206 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Aug 12 02:37:09 dist-test-slave-rmhg krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502229, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 02:37:09 dist-test-slave-rmhg krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502229, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.127.129@KRBTEST.COM
I20260812 02:37:09.894189 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:09.884173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46860 (local address 127.3.127.129:36221)
0812 02:37:09.884390 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:09.884395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:09.884585 (+ 190us) server_negotiation.cc:407] Connection header received
0812 02:37:09.884759 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:09.884762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:09.884812 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:09.884886 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:09.886038 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.886638 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.887532 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.887777 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.887874 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:09.890765 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:09.890769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:09.890771 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:09.890809 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:09.892266 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:09.892772 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:09.892775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:09.892777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:09.892869 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:09.893182 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:09.893186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:09.893188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:09.893405 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:09.893583 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:09.893871 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 02:37:09.894012 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
Aug 12 02:37:09 dist-test-slave-rmhg krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 02:37:09 dist-test-slave-rmhg krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.129@KRBTEST.COM
I20260812 02:37:09.925400 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:09.916363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46864 (local address 127.3.127.129:36221)
0812 02:37:09.916505 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:09.916510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:09.916676 (+ 166us) server_negotiation.cc:407] Connection header received
0812 02:37:09.916879 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:09.916882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:09.916933 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:09.917005 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:09.917834 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.918360 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.919251 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:09.919473 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:09.919568 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:09.922182 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:09.922186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:09.922188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:09.922219 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:09.923608 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:09.924079 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:09.924083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:09.924085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:09.924174 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:09.924441 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:09.924444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:09.924446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:09.924668 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:09.924822 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:09.925057 (+ 235us) server_negotiation.cc:299] Negotiation successful
0812 02:37:09.925234 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260812 02:37:09.927361 4685 tablet_service.cc:2908] Rejecting scan request for tablet 3446f4e45eae4aefb8e8d8b5408481ed: Uninitialized: clean time has not yet been initialized
I20260812 02:37:09.929986 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 4616
I20260812 02:37:09.942298 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 4848
I20260812 02:37:09.964192 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 5076
I20260812 02:37:09.983593 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 4471
2026-08-12T02:37:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2745 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:37:10 dist-test-slave-rmhg krb5kdc[5342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:37:10 dist-test-slave-rmhg krb5kdc[5342](info): set up 2 sockets
Aug 12 02:37:10 dist-test-slave-rmhg krb5kdc[5342](info): commencing operation
krb5kdc: starting...
W20260812 02:37:11.182602 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:37:11 dist-test-slave-rmhg krb5kdc[5342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:37:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:37:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:37:11.324533 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:41453
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:41453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:11.447240 5361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:11.447558 5361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:11.447623 5361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:11.451787 5361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:11.451867 5361 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 02:37:11.451894 5361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:11.451916 5361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:11.451939 5361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:11.451967 5361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:37:11.457747 5361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:41453
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:41453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:11.459054 5361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:11.460192 5361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:11.469848 5369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:11.470041 5361 server_base.cc:1034] running on GCE node
W20260812 02:37:11.469888 5372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:11.469888 5370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:11.470729 5361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:11.471971 5361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:11.473176 5361 hybrid_clock.cc:648] HybridClock initialized: now 1786502231473123 us; error 44 us; skew 500 ppm
Aug 12 02:37:11 dist-test-slave-rmhg krb5kdc[5342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502231, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:11.480806 5361 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:37:11.481973 5361 webserver.cc:460] Webserver started at http://127.3.127.190:44513/ using document root <none> and password file <none>
I20260812 02:37:11.482615 5361 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:11.482692 5361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:11.482923 5361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:11.485144 5361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "34ce3a8807e3477999428a2e306c9b25"
format_stamp: "Formatted at 2026-08-12 02:37:11 on dist-test-slave-rmhg"
server_key: "4977e4dd220afee49debe185adcb2a8f"
server_key_iv: "af3e2574daf16d58660b992ee9701f38"
server_key_version: "encryptionkey@0"
I20260812 02:37:11.485800 5361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "34ce3a8807e3477999428a2e306c9b25"
format_stamp: "Formatted at 2026-08-12 02:37:11 on dist-test-slave-rmhg"
server_key: "4977e4dd220afee49debe185adcb2a8f"
server_key_iv: "af3e2574daf16d58660b992ee9701f38"
server_key_version: "encryptionkey@0"
I20260812 02:37:11.490336 5361 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:37:11.493278 5380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:11.494751 5361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:11.494890 5361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "34ce3a8807e3477999428a2e306c9b25"
format_stamp: "Formatted at 2026-08-12 02:37:11 on dist-test-slave-rmhg"
server_key: "4977e4dd220afee49debe185adcb2a8f"
server_key_iv: "af3e2574daf16d58660b992ee9701f38"
server_key_version: "encryptionkey@0"
I20260812 02:37:11.495031 5361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:11.512398 5361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:11.515889 5361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:11.516083 5361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:11.524673 5361 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:41453
I20260812 02:37:11.524682 5481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:41453 every 8 connection(s)
I20260812 02:37:11.525701 5361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:37:11.528693 5482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:11.534185 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 5361
I20260812 02:37:11.534302 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:37:11.534536 3582 external_mini_cluster.cc:1428] Setting key 635dcef70820d4ceb7c1cbaf87e100a5
I20260812 02:37:11.534718 5482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25: Bootstrap starting.
I20260812 02:37:11.537425 5482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:11.538376 5482 log.cc:824] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:11.540690 5482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25: No bootstrap required, opened a new log
Aug 12 02:37:11 dist-test-slave-rmhg krb5kdc[5342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:11.545785 5482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ce3a8807e3477999428a2e306c9b25" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41453 } }
I20260812 02:37:11.546147 5482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:11.546219 5482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ce3a8807e3477999428a2e306c9b25, State: Initialized, Role: FOLLOWER
I20260812 02:37:11.546759 5482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ce3a8807e3477999428a2e306c9b25" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41453 } }
I20260812 02:37:11.546926 5482 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:37:11.547003 5482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:37:11.547114 5482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:11.548094 5482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ce3a8807e3477999428a2e306c9b25" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41453 } }
I20260812 02:37:11.548467 5482 leader_election.cc:302] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34ce3a8807e3477999428a2e306c9b25; no voters:
I20260812 02:37:11.548848 5482 leader_election.cc:288] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:37:11.549011 5494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:11.549269 5494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25 [term 1 LEADER]: Becoming Leader. State: Replica: 34ce3a8807e3477999428a2e306c9b25, State: Running, Role: LEADER
I20260812 02:37:11.549594 5494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ce3a8807e3477999428a2e306c9b25" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41453 } }
I20260812 02:37:11.550316 5482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:37:11.550323 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:11.535884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.3.127.190:41453)
0812 02:37:11.536332 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:11.536349 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:11.536392 (+ 43us) server_negotiation.cc:407] Connection header received
0812 02:37:11.537078 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:11.537097 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:11.537529 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:11.537867 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:11.538794 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:11.539920 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:11.541100 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:11.541655 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:11.541823 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:11.544775 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:11.544781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:11.544798 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:11.544832 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:11.546827 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:11.547390 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:11.547397 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:11.547403 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:11.547582 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:11.547880 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:11.547884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:11.547885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:11.548368 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:11.548562 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:11.548878 (+ 316us) server_negotiation.cc:299] Negotiation successful
0812 02:37:11.549410 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":301,"thread_start_us":142,"threads_started":1}
I20260812 02:37:11.551458 5497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34ce3a8807e3477999428a2e306c9b25. Latest consensus state: current_term: 1 leader_uuid: "34ce3a8807e3477999428a2e306c9b25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ce3a8807e3477999428a2e306c9b25" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41453 } } }
I20260812 02:37:11.551498 5495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34ce3a8807e3477999428a2e306c9b25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ce3a8807e3477999428a2e306c9b25" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41453 } } }
I20260812 02:37:11.551590 5497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:11.551604 5495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:11.552965 5507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:37:11.556145 5507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:37:11.562263 5507 catalog_manager.cc:1347] Generated new cluster ID: 93c3f8078c514a7581b256781cd451ac
I20260812 02:37:11.562338 5507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:37:11.571961 5507 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:37:11.573485 5507 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:37:11.583201 5507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25: Generated new TSK 0
I20260812 02:37:11.584249 5507 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
I20260812 02:37:11.622638 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:41453
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:11.746552 5527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:11.746873 5527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:11.746920 5527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:11.751257 5527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:11.751339 5527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:11.751382 5527 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:37:11.751421 5527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:37:11.757624 5527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.127.190:41453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:11.758864 5527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:11.759989 5527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:11.770833 5537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:11.770833 5540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:11.770844 5538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:11.771276 5527 server_base.cc:1034] running on GCE node
I20260812 02:37:11.771746 5527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:11.772469 5527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:11.773669 5527 hybrid_clock.cc:648] HybridClock initialized: now 1786502231773641 us; error 43 us; skew 500 ppm
Aug 12 02:37:11 dist-test-slave-rmhg krb5kdc[5342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502231, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:11.776865 5527 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:37:11.778059 5527 webserver.cc:460] Webserver started at http://127.3.127.129:46835/ using document root <none> and password file <none>
I20260812 02:37:11.778673 5527 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:11.778744 5527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:11.778972 5527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:11.781320 5527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "220d893d0a444e069882aa6f3a1269af"
format_stamp: "Formatted at 2026-08-12 02:37:11 on dist-test-slave-rmhg"
server_key: "46a5c8ab4d18bc171578b888d374b74e"
server_key_iv: "b3da1fe0846f9eabde87e0ef78dcfa17"
server_key_version: "encryptionkey@0"
I20260812 02:37:11.781944 5527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "220d893d0a444e069882aa6f3a1269af"
format_stamp: "Formatted at 2026-08-12 02:37:11 on dist-test-slave-rmhg"
server_key: "46a5c8ab4d18bc171578b888d374b74e"
server_key_iv: "b3da1fe0846f9eabde87e0ef78dcfa17"
server_key_version: "encryptionkey@0"
I20260812 02:37:11.786270 5527 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260812 02:37:11.789127 5553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:11.790750 5527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:37:11.790900 5527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "220d893d0a444e069882aa6f3a1269af"
format_stamp: "Formatted at 2026-08-12 02:37:11 on dist-test-slave-rmhg"
server_key: "46a5c8ab4d18bc171578b888d374b74e"
server_key_iv: "b3da1fe0846f9eabde87e0ef78dcfa17"
server_key_version: "encryptionkey@0"
I20260812 02:37:11.791038 5527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:11.806572 5527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:11.809993 5527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:11.810223 5527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:11.811831 5527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:11.811890 5527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:11.811947 5527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:11.811976 5527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:37:11 dist-test-slave-rmhg krb5kdc[5342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502231, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:11.823207 5527 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:46309
I20260812 02:37:11.823256 5740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:46309 every 8 connection(s)
I20260812 02:37:11.824286 5527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:37:11.827724 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:11.812792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:32997 (local address 127.3.127.190:41453)
0812 02:37:11.812914 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:11.812918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:11.814112 (+ 1194us) server_negotiation.cc:407] Connection header received
0812 02:37:11.815292 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:11.815297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:11.815363 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:11.815432 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:11.817391 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:11.818063 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:11.819947 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:11.820284 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:11.820383 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:11.823790 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:11.823794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:11.823796 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:11.823826 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:11.825398 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:11.826046 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:11.826050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:11.826052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:11.826142 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:11.826496 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:11.826500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:11.826502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:11.826720 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:11.826815 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:11.827373 (+ 558us) server_negotiation.cc:299] Negotiation successful
0812 02:37:11.827548 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 02:37:11.828574 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:11.813403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41453 (local address 127.3.127.129:32997)
0812 02:37:11.813936 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:11.813984 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:11.814993 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:11.815928 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:11.815935 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:11.816408 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:11.817197 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:11.817223 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:11.818202 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:11.818205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:11.819770 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:11.819784 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:11.820468 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:11.820472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:11.820770 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:11.821519 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:11.821543 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:11.823577 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:11.825533 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:11.825540 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:11.825557 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:11.825926 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:11.826230 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:11.826233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:11.826235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:11.826389 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:11.826829 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:11.826837 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:11.827113 (+ 276us) client_negotiation.cc:769] Sending connection context
0812 02:37:11.827292 (+ 179us) client_negotiation.cc:240] Negotiation successful
0812 02:37:11.827668 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":129,"threads_started":1}
I20260812 02:37:11.830061 5741 heartbeater.cc:347] Connected to a master server at 127.3.127.190:41453
I20260812 02:37:11.830230 5741 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:11.830799 5741 heartbeater.cc:511] Master 127.3.127.190:41453 requested a full tablet report, sending...
I20260812 02:37:11.831311 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 5527
I20260812 02:37:11.831425 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:37:11.831697 3582 external_mini_cluster.cc:1428] Setting key 6c8fe2816732963d3f5292a2f95e9d64
I20260812 02:37:11.833379 5413 ts_manager.cc:194] Registered new tserver with Master: 220d893d0a444e069882aa6f3a1269af (127.3.127.129:46309)
I20260812 02:37:11.834573 5413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:32997
I20260812 02:37:11.842659 5756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:11.832691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41453 (local address 127.3.127.129:40753)
0812 02:37:11.835260 (+ 2569us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:11.835279 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:11.835404 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:11.835662 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:11.835665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:11.835910 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:11.836035 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:11.836042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:11.836830 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:11.836834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:11.837487 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:11.837498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:11.837939 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:11.837943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:11.838048 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:11.838716 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:11.838730 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:11.839109 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:11.840944 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:11.840948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:11.840950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:11.841249 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:11.841542 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:11.841546 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:11.841548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:11.841642 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:11.842053 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:11.842056 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:11.842173 (+ 117us) client_negotiation.cc:769] Sending connection context
0812 02:37:11.842261 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 02:37:11.842443 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":153,"threads_started":1}
I20260812 02:37:11.842654 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:11.832761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:40753 (local address 127.3.127.190:41453)
0812 02:37:11.832883 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:11.832887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:11.835364 (+ 2477us) server_negotiation.cc:407] Connection header received
0812 02:37:11.835496 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:11.835499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:11.835550 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:11.835626 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:11.836158 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:11.836697 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:11.837607 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:11.837826 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:11.837918 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:11.839281 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:11.839284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:11.839286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:11.839312 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:11.840802 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:11.841353 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:11.841356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:11.841357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:11.841447 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:11.841739 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:11.841742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:11.841743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:11.841952 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:11.842043 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:11.842353 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 02:37:11.842476 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.3.127.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
I20260812 02:37:11.864403 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:41453
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:11.983449 5760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:11.983765 5760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:11.983832 5760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:11.988045 5760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:11.988130 5760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:11.988189 5760 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:37:11.988247 5760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:37:11.994086 5760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.127.190:41453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:11.995311 5760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:11.996466 5760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:12.006924 5780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:12.006922 5775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:12.006916 5776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:12.007170 5760 server_base.cc:1034] running on GCE node
I20260812 02:37:12.007736 5760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:12.008460 5760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:12.009639 5760 hybrid_clock.cc:648] HybridClock initialized: now 1786502232009601 us; error 59 us; skew 500 ppm
Aug 12 02:37:12 dist-test-slave-rmhg krb5kdc[5342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502232, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:12.012768 5760 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:37:12.013906 5760 webserver.cc:460] Webserver started at http://127.3.127.130:44553/ using document root <none> and password file <none>
I20260812 02:37:12.014540 5760 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:12.014619 5760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:12.014842 5760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:12.017100 5760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "ec5174c38bbe431bb178024d344f137a"
format_stamp: "Formatted at 2026-08-12 02:37:12 on dist-test-slave-rmhg"
server_key: "5a187787690d49be49ef98a0c882286b"
server_key_iv: "4a1cf514ddb5713385c8d26b0c77a8b7"
server_key_version: "encryptionkey@0"
I20260812 02:37:12.017915 5760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "ec5174c38bbe431bb178024d344f137a"
format_stamp: "Formatted at 2026-08-12 02:37:12 on dist-test-slave-rmhg"
server_key: "5a187787690d49be49ef98a0c882286b"
server_key_iv: "4a1cf514ddb5713385c8d26b0c77a8b7"
server_key_version: "encryptionkey@0"
I20260812 02:37:12.022661 5760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:37:12.025736 5793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:12.027271 5760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:12.027437 5760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "ec5174c38bbe431bb178024d344f137a"
format_stamp: "Formatted at 2026-08-12 02:37:12 on dist-test-slave-rmhg"
server_key: "5a187787690d49be49ef98a0c882286b"
server_key_iv: "4a1cf514ddb5713385c8d26b0c77a8b7"
server_key_version: "encryptionkey@0"
I20260812 02:37:12.027571 5760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:12.048477 5760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:12.052206 5760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:12.052407 5760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:12.054118 5760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:12.054180 5760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:12.054239 5760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:12.054270 5760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:37:12 dist-test-slave-rmhg krb5kdc[5342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502232, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:12.065563 5760 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:36409
I20260812 02:37:12.065585 5981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:36409 every 8 connection(s)
I20260812 02:37:12.066771 5760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:37:12.068207 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.054965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:58533 (local address 127.3.127.190:41453)
0812 02:37:12.055130 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:12.055136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:12.056129 (+ 993us) server_negotiation.cc:407] Connection header received
0812 02:37:12.057420 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:12.057425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:12.057541 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:12.057647 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:12.059089 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.059638 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.060908 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.061222 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.061327 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.064163 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:12.064168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:12.064171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:12.064205 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:12.066009 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:12.066535 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:12.066538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:12.066539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:12.066626 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:12.066961 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:12.066965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:12.066967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:12.067248 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:12.067348 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:12.067954 (+ 606us) server_negotiation.cc:299] Negotiation successful
0812 02:37:12.068071 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 02:37:12.069097 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.055394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41453 (local address 127.3.127.130:58533)
0812 02:37:12.055918 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:12.055985 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:12.057125 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:12.057882 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:12.057890 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:12.058325 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:12.058918 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.058934 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.059771 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.059775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.060777 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.060789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.061396 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.061400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.061615 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.062322 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:12.062342 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:12.063957 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:12.066137 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:12.066142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:12.066155 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:12.066422 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:12.066717 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:12.066720 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:12.066722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:12.066864 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:12.067409 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:12.067416 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:12.067719 (+ 303us) client_negotiation.cc:769] Sending connection context
0812 02:37:12.067887 (+ 168us) client_negotiation.cc:240] Negotiation successful
0812 02:37:12.068172 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":114,"threads_started":1}
I20260812 02:37:12.070485 5982 heartbeater.cc:347] Connected to a master server at 127.3.127.190:41453
I20260812 02:37:12.070676 5982 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:12.071237 5982 heartbeater.cc:511] Master 127.3.127.190:41453 requested a full tablet report, sending...
I20260812 02:37:12.072448 5413 ts_manager.cc:194] Registered new tserver with Master: ec5174c38bbe431bb178024d344f137a (127.3.127.130:36409)
I20260812 02:37:12.072734 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 5760
I20260812 02:37:12.072849 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:37:12.073215 3582 external_mini_cluster.cc:1428] Setting key 70325dad4327639463c5b28ae2a80241
I20260812 02:37:12.073262 5413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:58533
I20260812 02:37:12.080961 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.073008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:57779 (local address 127.3.127.190:41453)
0812 02:37:12.073281 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:12.073286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:12.073818 (+ 532us) server_negotiation.cc:407] Connection header received
0812 02:37:12.073983 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:12.073987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:12.074039 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:12.074137 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:12.074695 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.075254 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.076062 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.076286 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.076374 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.077665 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:12.077668 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:12.077670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:12.077697 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:12.079117 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:12.079625 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:12.079628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:12.079630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:12.079743 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:12.080056 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:12.080059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:12.080060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:12.080307 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:12.080389 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:12.080700 (+ 311us) server_negotiation.cc:299] Negotiation successful
0812 02:37:12.080824 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":193}
I20260812 02:37:12.081049 5989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.073043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41453 (local address 127.3.127.130:57779)
0812 02:37:12.073734 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:12.073748 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:12.073857 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:12.074185 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:12.074188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:12.074435 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:12.074575 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.074585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.075380 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.075383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.075951 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.075960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.076390 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.076394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.076485 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.076996 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:12.077010 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:12.077495 (+ 485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:12.079249 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:12.079253 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:12.079255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:12.079525 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:12.079833 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:12.079836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:12.079838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:12.079973 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:12.080409 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:12.080412 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:12.080536 (+ 124us) client_negotiation.cc:769] Sending connection context
0812 02:37:12.080656 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 02:37:12.080834 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":145,"threads_started":1}
WARNING: no policy specified for kudu/127.3.127.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
I20260812 02:37:12.104426 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:41453
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:12.227864 5994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:12.228186 5994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:12.228237 5994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:12.232785 5994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:12.232864 5994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:12.232906 5994 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:37:12.232939 5994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:37:12.238819 5994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.127.190:41453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:12.240013 5994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:12.241112 5994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:12.251747 6002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:12.251758 6001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:12.251889 6004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:12.252221 5994 server_base.cc:1034] running on GCE node
I20260812 02:37:12.252724 5994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:12.253479 5994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:12.254675 5994 hybrid_clock.cc:648] HybridClock initialized: now 1786502232254646 us; error 40 us; skew 500 ppm
Aug 12 02:37:12 dist-test-slave-rmhg krb5kdc[5342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502232, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:12.257695 5994 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:37:12.258802 5994 webserver.cc:460] Webserver started at http://127.3.127.131:41211/ using document root <none> and password file <none>
I20260812 02:37:12.259440 5994 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:12.259513 5994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:12.259722 5994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:12.262090 5994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "76c5ade3cbce49ec9e84c3d8dab52867"
format_stamp: "Formatted at 2026-08-12 02:37:12 on dist-test-slave-rmhg"
server_key: "0f6b1d85c5ee5e4412093222a6f0d1ca"
server_key_iv: "09d084bb75862fe2ab974de5bbb48332"
server_key_version: "encryptionkey@0"
I20260812 02:37:12.262732 5994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "76c5ade3cbce49ec9e84c3d8dab52867"
format_stamp: "Formatted at 2026-08-12 02:37:12 on dist-test-slave-rmhg"
server_key: "0f6b1d85c5ee5e4412093222a6f0d1ca"
server_key_iv: "09d084bb75862fe2ab974de5bbb48332"
server_key_version: "encryptionkey@0"
I20260812 02:37:12.266966 5994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:37:12.269894 6015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:12.271350 5994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:12.271481 5994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "76c5ade3cbce49ec9e84c3d8dab52867"
format_stamp: "Formatted at 2026-08-12 02:37:12 on dist-test-slave-rmhg"
server_key: "0f6b1d85c5ee5e4412093222a6f0d1ca"
server_key_iv: "09d084bb75862fe2ab974de5bbb48332"
server_key_version: "encryptionkey@0"
I20260812 02:37:12.271611 5994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:12.312474 5994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:12.316077 5994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:12.316286 5994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:12.318066 5994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:12.318132 5994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:12.318190 5994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:12.318221 5994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:37:12 dist-test-slave-rmhg krb5kdc[5342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502232, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:12.329254 5994 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:39671
I20260812 02:37:12.329275 6191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:39671 every 8 connection(s)
I20260812 02:37:12.330526 5994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:37:12.333391 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 5994
I20260812 02:37:12.333500 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:37:12.333819 3582 external_mini_cluster.cc:1428] Setting key 254137afefc4746e382318088cdafbe0
I20260812 02:37:12.334088 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.319156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:44641 (local address 127.3.127.190:41453)
0812 02:37:12.319294 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:12.319298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:12.320596 (+ 1298us) server_negotiation.cc:407] Connection header received
0812 02:37:12.321858 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:12.321862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:12.321925 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:12.322009 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:12.323892 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.324833 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.326684 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.326923 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.327026 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.330012 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:12.330017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:12.330020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:12.330056 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:12.331759 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:12.332324 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:12.332328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:12.332330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:12.332428 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:12.332830 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:12.332832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:12.332834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:12.333045 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:12.333128 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:12.333785 (+ 657us) server_negotiation.cc:299] Negotiation successful
0812 02:37:12.333929 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 02:37:12.335095 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.319782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41453 (local address 127.3.127.131:44641)
0812 02:37:12.320404 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:12.320457 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:12.321613 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:12.322267 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:12.322276 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:12.322818 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:12.323644 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.323671 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.325019 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.325022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.326533 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.326547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.327110 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.327116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.327311 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.327973 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:12.327996 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:12.329782 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:12.331901 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:12.331907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:12.331922 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:12.332205 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:12.332519 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:12.332521 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:12.332522 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:12.332725 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:12.333183 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:12.333191 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:12.333480 (+ 289us) client_negotiation.cc:769] Sending connection context
0812 02:37:12.333710 (+ 230us) client_negotiation.cc:240] Negotiation successful
0812 02:37:12.334043 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":182,"threads_started":1}
I20260812 02:37:12.337121 6192 heartbeater.cc:347] Connected to a master server at 127.3.127.190:41453
I20260812 02:37:12.337337 6192 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:12.338068 6192 heartbeater.cc:511] Master 127.3.127.190:41453 requested a full tablet report, sending...
I20260812 02:37:12.339545 5414 ts_manager.cc:194] Registered new tserver with Master: 76c5ade3cbce49ec9e84c3d8dab52867 (127.3.127.131:39671)
I20260812 02:37:12.340304 5414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:44641
I20260812 02:37:12.347275 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.340075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:38591 (local address 127.3.127.190:41453)
0812 02:37:12.340219 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:12.340223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:12.340442 (+ 219us) server_negotiation.cc:407] Connection header received
0812 02:37:12.340593 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:12.340596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:12.340636 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:12.340715 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:12.341280 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.341796 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.342629 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.342862 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.342955 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.344071 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:12.344074 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:12.344076 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:12.344101 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:12.345466 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:12.345894 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:12.345897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:12.345898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:12.345984 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:12.346269 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:12.346272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:12.346274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:12.346524 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:12.346633 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:12.346999 (+ 366us) server_negotiation.cc:299] Negotiation successful
0812 02:37:12.347138 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 02:37:12.347289 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.339970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41453 (local address 127.3.127.131:38591)
0812 02:37:12.340362 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:12.340376 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:12.340491 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:12.340767 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:12.340770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:12.340973 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:12.341131 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.341139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.341935 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.341938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.342511 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.342520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.342971 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.342975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.343074 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.343566 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:12.343578 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:12.343925 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:12.345576 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:12.345579 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:12.345582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:12.345798 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:12.346091 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:12.346094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:12.346097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:12.346183 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:12.346664 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:12.346668 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:12.346794 (+ 126us) client_negotiation.cc:769] Sending connection context
0812 02:37:12.346903 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 02:37:12.347077 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":2,"thread_start_us":105,"threads_started":1}
I20260812 02:37:12.348464 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 02:37:12 dist-test-slave-rmhg krb5kdc[5342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502232, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 02:37:12 dist-test-slave-rmhg krb5kdc[5342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502232, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:12.377807 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.368787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33648 (local address 127.3.127.190:41453)
0812 02:37:12.368951 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:12.368954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:12.368967 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:37:12.369012 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:12.369014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:12.369057 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:12.369123 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:12.369926 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.370522 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.371434 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.371684 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.371773 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.374382 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:12.374385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:12.374387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:12.374421 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:12.375929 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:12.376415 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:12.376419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:12.376421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:12.376519 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:12.376856 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:12.376859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:12.376861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:12.377082 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:12.377248 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:12.377501 (+ 253us) server_negotiation.cc:299] Negotiation successful
0812 02:37:12.377628 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 02:37:12.382982 5414 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:37:12.386673 5414 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:37:12.402844 6217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.396428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:46309 (local address 127.0.0.1:39382)
0812 02:37:12.397278 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:12.397345 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:12.397534 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:12.398189 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:12.398196 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:12.398228 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:12.398418 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.398427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.399847 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.399850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.401055 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.401067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.402253 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.402257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.402382 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.402437 (+ 55us) client_negotiation.cc:769] Sending connection context
0812 02:37:12.402532 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 02:37:12.402676 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":98,"threads_started":1}
I20260812 02:37:12.403393 6224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.397381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:39671 (local address 127.0.0.1:57596)
0812 02:37:12.398344 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:12.398359 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:12.398459 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:12.399026 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:12.399029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:12.399048 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:12.399139 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.399146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.400543 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.400547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.401748 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.401758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.402953 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.402957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.403066 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.403092 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 02:37:12.403157 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 02:37:12.403250 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"mutex_wait_us":2,"thread_start_us":114,"threads_started":1}
I20260812 02:37:12.403595 6219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.396567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39382 (local address 127.3.127.129:46309)
0812 02:37:12.396930 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:12.396937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:12.397445 (+ 508us) server_negotiation.cc:407] Connection header received
0812 02:37:12.397756 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:12.397763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:12.398013 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:12.398170 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:12.398583 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.399712 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.401254 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.402093 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.403085 (+ 992us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.403272 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:12.403324 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 02:37:12.403457 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":131,"threads_started":1}
I20260812 02:37:12.403843 6225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.397590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57596 (local address 127.3.127.131:39671)
0812 02:37:12.397967 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:12.397980 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:12.398461 (+ 481us) server_negotiation.cc:407] Connection header received
0812 02:37:12.398566 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:12.398571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:12.398736 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:12.398862 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:12.399273 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.400417 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.401883 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.402637 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.402724 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.402889 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:12.403601 (+ 712us) server_negotiation.cc:299] Negotiation successful
0812 02:37:12.403705 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":128,"threads_started":1}
I20260812 02:37:12.403847 6227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.397923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37828 (local address 127.3.127.130:36409)
0812 02:37:12.398286 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:12.398299 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:12.398323 (+ 24us) server_negotiation.cc:407] Connection header received
0812 02:37:12.398388 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:12.398391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:12.398595 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:12.398754 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:12.399200 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.400480 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.401611 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.402585 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.402701 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.403536 (+ 835us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:12.403598 (+ 62us) server_negotiation.cc:299] Negotiation successful
0812 02:37:12.403702 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":131,"threads_started":1}
I20260812 02:37:12.404125 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.396608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:36409 (local address 127.0.0.1:37828)
0812 02:37:12.397340 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:12.397356 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:12.397513 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:12.398745 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:12.398748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:12.398767 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:12.398867 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.398873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.400581 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.400585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.401480 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.401490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.403184 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.403190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.403272 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.403292 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 02:37:12.403348 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 02:37:12.403412 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":44,"thread_start_us":99,"threads_started":1}
I20260812 02:37:12.407035 6097 tablet_service.cc:1467] Processing CreateTablet for tablet e34db760421b4e06871b28d61d73b07c (DEFAULT_TABLE table=test-table [id=7c7b95a477ee42f684634af8845de140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:12.407040 5877 tablet_service.cc:1467] Processing CreateTablet for tablet e34db760421b4e06871b28d61d73b07c (DEFAULT_TABLE table=test-table [id=7c7b95a477ee42f684634af8845de140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:12.407039 5634 tablet_service.cc:1467] Processing CreateTablet for tablet e34db760421b4e06871b28d61d73b07c (DEFAULT_TABLE table=test-table [id=7c7b95a477ee42f684634af8845de140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:12.408165 6097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e34db760421b4e06871b28d61d73b07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:12.408159 5634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e34db760421b4e06871b28d61d73b07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:12.408159 5877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e34db760421b4e06871b28d61d73b07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:12.416142 6230 tablet_bootstrap.cc:492] T e34db760421b4e06871b28d61d73b07c P 220d893d0a444e069882aa6f3a1269af: Bootstrap starting.
I20260812 02:37:12.416581 6234 tablet_bootstrap.cc:492] T e34db760421b4e06871b28d61d73b07c P 76c5ade3cbce49ec9e84c3d8dab52867: Bootstrap starting.
I20260812 02:37:12.418756 6235 tablet_bootstrap.cc:492] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a: Bootstrap starting.
I20260812 02:37:12.419102 6234 tablet_bootstrap.cc:654] T e34db760421b4e06871b28d61d73b07c P 76c5ade3cbce49ec9e84c3d8dab52867: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:12.419389 6230 tablet_bootstrap.cc:654] T e34db760421b4e06871b28d61d73b07c P 220d893d0a444e069882aa6f3a1269af: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:12.420217 6230 log.cc:824] T e34db760421b4e06871b28d61d73b07c P 220d893d0a444e069882aa6f3a1269af: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:12.420217 6234 log.cc:824] T e34db760421b4e06871b28d61d73b07c P 76c5ade3cbce49ec9e84c3d8dab52867: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:12.420897 6235 tablet_bootstrap.cc:654] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:12.421819 6235 log.cc:824] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:12.422585 6230 tablet_bootstrap.cc:492] T e34db760421b4e06871b28d61d73b07c P 220d893d0a444e069882aa6f3a1269af: No bootstrap required, opened a new log
I20260812 02:37:12.422814 6230 ts_tablet_manager.cc:1397] T e34db760421b4e06871b28d61d73b07c P 220d893d0a444e069882aa6f3a1269af: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 02:37:12.422912 6234 tablet_bootstrap.cc:492] T e34db760421b4e06871b28d61d73b07c P 76c5ade3cbce49ec9e84c3d8dab52867: No bootstrap required, opened a new log
I20260812 02:37:12.423103 6234 ts_tablet_manager.cc:1397] T e34db760421b4e06871b28d61d73b07c P 76c5ade3cbce49ec9e84c3d8dab52867: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 02:37:12.423621 6235 tablet_bootstrap.cc:492] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a: No bootstrap required, opened a new log
I20260812 02:37:12.423794 6235 ts_tablet_manager.cc:1397] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260812 02:37:12.427040 6235 raft_consensus.cc:348] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.427312 6235 raft_consensus.cc:374] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:12.427398 6235 raft_consensus.cc:729] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec5174c38bbe431bb178024d344f137a, State: Initialized, Role: FOLLOWER
I20260812 02:37:12.427503 6234 raft_consensus.cc:348] T e34db760421b4e06871b28d61d73b07c P 76c5ade3cbce49ec9e84c3d8dab52867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.427814 6234 raft_consensus.cc:374] T e34db760421b4e06871b28d61d73b07c P 76c5ade3cbce49ec9e84c3d8dab52867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:12.427882 6234 raft_consensus.cc:729] T e34db760421b4e06871b28d61d73b07c P 76c5ade3cbce49ec9e84c3d8dab52867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c5ade3cbce49ec9e84c3d8dab52867, State: Initialized, Role: FOLLOWER
I20260812 02:37:12.427825 6230 raft_consensus.cc:348] T e34db760421b4e06871b28d61d73b07c P 220d893d0a444e069882aa6f3a1269af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.427959 6235 consensus_queue.cc:260] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.428072 6230 raft_consensus.cc:374] T e34db760421b4e06871b28d61d73b07c P 220d893d0a444e069882aa6f3a1269af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:12.428119 6230 raft_consensus.cc:729] T e34db760421b4e06871b28d61d73b07c P 220d893d0a444e069882aa6f3a1269af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220d893d0a444e069882aa6f3a1269af, State: Initialized, Role: FOLLOWER
I20260812 02:37:12.428593 6230 consensus_queue.cc:260] T e34db760421b4e06871b28d61d73b07c P 220d893d0a444e069882aa6f3a1269af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.428593 6234 consensus_queue.cc:260] T e34db760421b4e06871b28d61d73b07c P 76c5ade3cbce49ec9e84c3d8dab52867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.429119 6235 ts_tablet_manager.cc:1428] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260812 02:37:12.429417 5741 heartbeater.cc:503] Master 127.3.127.190:41453 was elected leader, sending a full tablet report...
I20260812 02:37:12.429718 6230 ts_tablet_manager.cc:1428] T e34db760421b4e06871b28d61d73b07c P 220d893d0a444e069882aa6f3a1269af: Time spent starting tablet: real 0.007s user 0.002s sys 0.006s
I20260812 02:37:12.429837 6192 heartbeater.cc:503] Master 127.3.127.190:41453 was elected leader, sending a full tablet report...
I20260812 02:37:12.430107 6234 ts_tablet_manager.cc:1428] T e34db760421b4e06871b28d61d73b07c P 76c5ade3cbce49ec9e84c3d8dab52867: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 02:37:12.430347 5982 heartbeater.cc:503] Master 127.3.127.190:41453 was elected leader, sending a full tablet report...
I20260812 02:37:12.453480 6239 raft_consensus.cc:482] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:37:12.453727 6239 raft_consensus.cc:504] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.454936 6239 leader_election.cc:288] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 220d893d0a444e069882aa6f3a1269af (127.3.127.129:46309), 76c5ade3cbce49ec9e84c3d8dab52867 (127.3.127.131:39671)
I20260812 02:37:12.459724 6246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.455270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:39671 (local address 127.3.127.130:51311)
0812 02:37:12.455716 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:12.455728 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:12.455814 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:12.456024 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:12.456027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:12.456040 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:12.456160 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.456167 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.457190 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.457194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.458575 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.458591 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.459336 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.459338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.459424 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.459448 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 02:37:12.459506 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 02:37:12.459570 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":22,"thread_start_us":139,"threads_started":1}
I20260812 02:37:12.459753 6225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.455381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:51311 (local address 127.3.127.131:39671)
0812 02:37:12.455503 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:12.455507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:12.455815 (+ 308us) server_negotiation.cc:407] Connection header received
0812 02:37:12.455881 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:12.455884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:12.455931 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:12.456002 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:12.456270 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.457017 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.458746 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.459199 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.459337 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.459400 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:12.459536 (+ 136us) server_negotiation.cc:299] Negotiation successful
0812 02:37:12.459617 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 02:37:12.460506 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.455241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:46309 (local address 127.3.127.130:45659)
0812 02:37:12.455452 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:12.455467 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:12.455573 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:12.455900 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:12.455902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:12.455917 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:12.456038 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.456045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.457190 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.457194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.459171 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.459181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.460076 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.460080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.460201 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.460227 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 02:37:12.460305 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 02:37:12.460380 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":4}
I20260812 02:37:12.460505 6126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e34db760421b4e06871b28d61d73b07c" candidate_uuid: "ec5174c38bbe431bb178024d344f137a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" is_pre_election: true
I20260812 02:37:12.460794 6219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.455366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:45659 (local address 127.3.127.129:46309)
0812 02:37:12.455505 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:12.455509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:12.455527 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:37:12.455720 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:12.455723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:12.455774 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:12.455863 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:12.456161 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.456984 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.459307 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.459963 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.460096 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.460149 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:12.460541 (+ 392us) server_negotiation.cc:299] Negotiation successful
0812 02:37:12.460634 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 02:37:12.460906 6126 raft_consensus.cc:2454] T e34db760421b4e06871b28d61d73b07c P 76c5ade3cbce49ec9e84c3d8dab52867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec5174c38bbe431bb178024d344f137a in term 0.
I20260812 02:37:12.461470 5797 leader_election.cc:302] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c5ade3cbce49ec9e84c3d8dab52867, ec5174c38bbe431bb178024d344f137a; no voters:
I20260812 02:37:12.461681 5665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e34db760421b4e06871b28d61d73b07c" candidate_uuid: "ec5174c38bbe431bb178024d344f137a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220d893d0a444e069882aa6f3a1269af" is_pre_election: true
I20260812 02:37:12.461812 6239 raft_consensus.cc:2781] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:37:12.461930 6239 raft_consensus.cc:482] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:37:12.461997 6239 raft_consensus.cc:3037] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:12.462000 5665 raft_consensus.cc:2454] T e34db760421b4e06871b28d61d73b07c P 220d893d0a444e069882aa6f3a1269af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec5174c38bbe431bb178024d344f137a in term 0.
I20260812 02:37:12.463210 6239 raft_consensus.cc:504] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.463570 6239 leader_election.cc:288] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a [CANDIDATE]: Term 1 election: Requested vote from peers 220d893d0a444e069882aa6f3a1269af (127.3.127.129:46309), 76c5ade3cbce49ec9e84c3d8dab52867 (127.3.127.131:39671)
I20260812 02:37:12.463953 5665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e34db760421b4e06871b28d61d73b07c" candidate_uuid: "ec5174c38bbe431bb178024d344f137a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220d893d0a444e069882aa6f3a1269af"
I20260812 02:37:12.463953 6126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e34db760421b4e06871b28d61d73b07c" candidate_uuid: "ec5174c38bbe431bb178024d344f137a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c5ade3cbce49ec9e84c3d8dab52867"
I20260812 02:37:12.464083 5665 raft_consensus.cc:3037] T e34db760421b4e06871b28d61d73b07c P 220d893d0a444e069882aa6f3a1269af [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:12.464083 6126 raft_consensus.cc:3037] T e34db760421b4e06871b28d61d73b07c P 76c5ade3cbce49ec9e84c3d8dab52867 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:12.465512 6126 raft_consensus.cc:2454] T e34db760421b4e06871b28d61d73b07c P 76c5ade3cbce49ec9e84c3d8dab52867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec5174c38bbe431bb178024d344f137a in term 1.
I20260812 02:37:12.465510 5665 raft_consensus.cc:2454] T e34db760421b4e06871b28d61d73b07c P 220d893d0a444e069882aa6f3a1269af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec5174c38bbe431bb178024d344f137a in term 1.
I20260812 02:37:12.465860 5797 leader_election.cc:302] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c5ade3cbce49ec9e84c3d8dab52867, ec5174c38bbe431bb178024d344f137a; no voters:
I20260812 02:37:12.466085 6239 raft_consensus.cc:2781] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:12.466327 6239 raft_consensus.cc:686] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a [term 1 LEADER]: Becoming Leader. State: Replica: ec5174c38bbe431bb178024d344f137a, State: Running, Role: LEADER
I20260812 02:37:12.466737 6239 consensus_queue.cc:237] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.470008 5414 catalog_manager.cc:5495] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a reported cstate change: term changed from 0 to 1, leader changed from <none> to ec5174c38bbe431bb178024d344f137a (127.3.127.130). New cstate: current_term: 1 leader_uuid: "ec5174c38bbe431bb178024d344f137a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } health_report { overall_health: UNKNOWN } } }
I20260812 02:37:12.495280 6227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.490899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37836 (local address 127.3.127.130:36409)
0812 02:37:12.491055 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:12.491063 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:12.491084 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:37:12.491456 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:12.491460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:12.491548 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:12.491677 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:12.492008 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.492563 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.493500 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.493773 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.493868 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.494155 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:12.494685 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:12.494784 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:12.495057 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 02:37:12.495153 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 02:37:12.502738 5665 raft_consensus.cc:1261] T e34db760421b4e06871b28d61d73b07c P 220d893d0a444e069882aa6f3a1269af [term 1 FOLLOWER]: Refusing update from remote peer ec5174c38bbe431bb178024d344f137a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:12.503666 6247 consensus_queue.cc:1035] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a [LEADER]: Connected to new peer: Peer: permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:12.504927 6126 raft_consensus.cc:1261] T e34db760421b4e06871b28d61d73b07c P 76c5ade3cbce49ec9e84c3d8dab52867 [term 1 FOLLOWER]: Refusing update from remote peer ec5174c38bbe431bb178024d344f137a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:12.505630 6247 consensus_queue.cc:1035] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:12.514088 6256 mvcc.cc:205] Tried to move back new op lower bound from 7317513144320925696 to 7317513144186621952. Current Snapshot: MvccSnapshot[applied={T|T < 7317513144320925696}]
I20260812 02:37:12.515084 6254 mvcc.cc:205] Tried to move back new op lower bound from 7317513144320925696 to 7317513144186621952. Current Snapshot: MvccSnapshot[applied={T|T < 7317513144320925696}]
I20260812 02:37:12.528321 5413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:12.528579 5413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:12.531487 5413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25: Sending DeleteTablet for 3 replicas of tablet e34db760421b4e06871b28d61d73b07c
I20260812 02:37:12.532615 5877 tablet_service.cc:1514] Processing DeleteTablet for tablet e34db760421b4e06871b28d61d73b07c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:12 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:37828
I20260812 02:37:12.532577 5634 tablet_service.cc:1514] Processing DeleteTablet for tablet e34db760421b4e06871b28d61d73b07c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:12 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:39382
I20260812 02:37:12.533190 6267 tablet_replica.cc:330] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a: stopping tablet replica
I20260812 02:37:12.533713 6097 tablet_service.cc:1514] Processing DeleteTablet for tablet e34db760421b4e06871b28d61d73b07c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:12 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:57596
I20260812 02:37:12.534328 6272 tablet_replica.cc:330] T e34db760421b4e06871b28d61d73b07c P 76c5ade3cbce49ec9e84c3d8dab52867: stopping tablet replica
I20260812 02:37:12.534752 6272 raft_consensus.cc:2229] T e34db760421b4e06871b28d61d73b07c P 76c5ade3cbce49ec9e84c3d8dab52867 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:12.535226 6272 raft_consensus.cc:2258] T e34db760421b4e06871b28d61d73b07c P 76c5ade3cbce49ec9e84c3d8dab52867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:12.535472 5413 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:37:12.535948 5413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:37:12.536738 6267 raft_consensus.cc:2229] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:37:12.537585 6267 raft_consensus.cc:2258] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:12.539273 6267 ts_tablet_manager.cc:1905] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:12.543574 5634 tablet_service.cc:1467] Processing CreateTablet for tablet c094f84039fc44a78f11a933ba7be6be (DEFAULT_TABLE table=test-table [id=7192696e8bcb40fcbaaed4588bf27ace]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:12.543939 5634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c094f84039fc44a78f11a933ba7be6be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:12.543947 5877 tablet_service.cc:1467] Processing CreateTablet for tablet c094f84039fc44a78f11a933ba7be6be (DEFAULT_TABLE table=test-table [id=7192696e8bcb40fcbaaed4588bf27ace]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:12.544253 5877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c094f84039fc44a78f11a933ba7be6be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:12.546105 6267 ts_tablet_manager.cc:1918] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:37:12.546224 6267 log.cc:1196] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/e34db760421b4e06871b28d61d73b07c
I20260812 02:37:12.547322 6230 tablet_bootstrap.cc:492] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af: Bootstrap starting.
I20260812 02:37:12.547479 6235 tablet_bootstrap.cc:492] T c094f84039fc44a78f11a933ba7be6be P ec5174c38bbe431bb178024d344f137a: Bootstrap starting.
I20260812 02:37:12.547890 6097 tablet_service.cc:1467] Processing CreateTablet for tablet c094f84039fc44a78f11a933ba7be6be (DEFAULT_TABLE table=test-table [id=7192696e8bcb40fcbaaed4588bf27ace]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:12.548210 6097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c094f84039fc44a78f11a933ba7be6be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:12.549055 6230 tablet_bootstrap.cc:654] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:12.549602 6235 tablet_bootstrap.cc:654] T c094f84039fc44a78f11a933ba7be6be P ec5174c38bbe431bb178024d344f137a: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:12.551111 6267 ts_tablet_manager.cc:1939] T e34db760421b4e06871b28d61d73b07c P ec5174c38bbe431bb178024d344f137a: Deleting consensus metadata
I20260812 02:37:12.551543 6268 tablet_replica.cc:330] T e34db760421b4e06871b28d61d73b07c P 220d893d0a444e069882aa6f3a1269af: stopping tablet replica
I20260812 02:37:12.551613 6234 tablet_bootstrap.cc:492] T c094f84039fc44a78f11a933ba7be6be P 76c5ade3cbce49ec9e84c3d8dab52867: Bootstrap starting.
I20260812 02:37:12.552196 6268 raft_consensus.cc:2229] T e34db760421b4e06871b28d61d73b07c P 220d893d0a444e069882aa6f3a1269af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:12.552665 6268 raft_consensus.cc:2258] T e34db760421b4e06871b28d61d73b07c P 220d893d0a444e069882aa6f3a1269af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:12.553517 6234 tablet_bootstrap.cc:654] T c094f84039fc44a78f11a933ba7be6be P 76c5ade3cbce49ec9e84c3d8dab52867: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:12.554536 6268 ts_tablet_manager.cc:1905] T e34db760421b4e06871b28d61d73b07c P 220d893d0a444e069882aa6f3a1269af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:12.554662 6235 tablet_bootstrap.cc:492] T c094f84039fc44a78f11a933ba7be6be P ec5174c38bbe431bb178024d344f137a: No bootstrap required, opened a new log
I20260812 02:37:12.554764 6235 ts_tablet_manager.cc:1397] T c094f84039fc44a78f11a933ba7be6be P ec5174c38bbe431bb178024d344f137a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260812 02:37:12.554800 5384 catalog_manager.cc:4841] TS ec5174c38bbe431bb178024d344f137a (127.3.127.130:36409): tablet e34db760421b4e06871b28d61d73b07c (table test-table [id=7c7b95a477ee42f684634af8845de140]) successfully deleted
I20260812 02:37:12.554922 6230 tablet_bootstrap.cc:492] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af: No bootstrap required, opened a new log
I20260812 02:37:12.555004 6230 ts_tablet_manager.cc:1397] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260812 02:37:12.555636 6235 raft_consensus.cc:348] T c094f84039fc44a78f11a933ba7be6be P ec5174c38bbe431bb178024d344f137a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.555764 6235 raft_consensus.cc:374] T c094f84039fc44a78f11a933ba7be6be P ec5174c38bbe431bb178024d344f137a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:12.555796 6235 raft_consensus.cc:729] T c094f84039fc44a78f11a933ba7be6be P ec5174c38bbe431bb178024d344f137a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec5174c38bbe431bb178024d344f137a, State: Initialized, Role: FOLLOWER
I20260812 02:37:12.555789 6230 raft_consensus.cc:348] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.555909 6230 raft_consensus.cc:374] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:12.555938 6230 raft_consensus.cc:729] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220d893d0a444e069882aa6f3a1269af, State: Initialized, Role: FOLLOWER
I20260812 02:37:12.555933 6235 consensus_queue.cc:260] T c094f84039fc44a78f11a933ba7be6be P ec5174c38bbe431bb178024d344f137a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.556077 6230 consensus_queue.cc:260] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.556346 6235 ts_tablet_manager.cc:1428] T c094f84039fc44a78f11a933ba7be6be P ec5174c38bbe431bb178024d344f137a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260812 02:37:12.556375 6230 ts_tablet_manager.cc:1428] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 02:37:12.557328 6272 ts_tablet_manager.cc:1905] T e34db760421b4e06871b28d61d73b07c P 76c5ade3cbce49ec9e84c3d8dab52867: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:12.558770 6268 ts_tablet_manager.cc:1918] T e34db760421b4e06871b28d61d73b07c P 220d893d0a444e069882aa6f3a1269af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:37:12.558979 6268 log.cc:1196] T e34db760421b4e06871b28d61d73b07c P 220d893d0a444e069882aa6f3a1269af: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-0/wal/wals/e34db760421b4e06871b28d61d73b07c
I20260812 02:37:12.559376 6268 ts_tablet_manager.cc:1939] T e34db760421b4e06871b28d61d73b07c P 220d893d0a444e069882aa6f3a1269af: Deleting consensus metadata
I20260812 02:37:12.561033 6272 ts_tablet_manager.cc:1918] T e34db760421b4e06871b28d61d73b07c P 76c5ade3cbce49ec9e84c3d8dab52867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:37:12.560966 5382 catalog_manager.cc:4841] TS 220d893d0a444e069882aa6f3a1269af (127.3.127.129:46309): tablet e34db760421b4e06871b28d61d73b07c (table test-table [id=7c7b95a477ee42f684634af8845de140]) successfully deleted
I20260812 02:37:12.561285 6272 log.cc:1196] T e34db760421b4e06871b28d61d73b07c P 76c5ade3cbce49ec9e84c3d8dab52867: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/e34db760421b4e06871b28d61d73b07c
I20260812 02:37:12.561872 6272 ts_tablet_manager.cc:1939] T e34db760421b4e06871b28d61d73b07c P 76c5ade3cbce49ec9e84c3d8dab52867: Deleting consensus metadata
I20260812 02:37:12.562018 6234 tablet_bootstrap.cc:492] T c094f84039fc44a78f11a933ba7be6be P 76c5ade3cbce49ec9e84c3d8dab52867: No bootstrap required, opened a new log
I20260812 02:37:12.562144 6234 ts_tablet_manager.cc:1397] T c094f84039fc44a78f11a933ba7be6be P 76c5ade3cbce49ec9e84c3d8dab52867: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260812 02:37:12.562724 6234 raft_consensus.cc:348] T c094f84039fc44a78f11a933ba7be6be P 76c5ade3cbce49ec9e84c3d8dab52867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.562844 6234 raft_consensus.cc:374] T c094f84039fc44a78f11a933ba7be6be P 76c5ade3cbce49ec9e84c3d8dab52867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:12.562873 6234 raft_consensus.cc:729] T c094f84039fc44a78f11a933ba7be6be P 76c5ade3cbce49ec9e84c3d8dab52867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c5ade3cbce49ec9e84c3d8dab52867, State: Initialized, Role: FOLLOWER
I20260812 02:37:12.563000 6234 consensus_queue.cc:260] T c094f84039fc44a78f11a933ba7be6be P 76c5ade3cbce49ec9e84c3d8dab52867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.563251 6234 ts_tablet_manager.cc:1428] T c094f84039fc44a78f11a933ba7be6be P 76c5ade3cbce49ec9e84c3d8dab52867: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:37:12.563521 5384 catalog_manager.cc:4841] TS 76c5ade3cbce49ec9e84c3d8dab52867 (127.3.127.131:39671): tablet e34db760421b4e06871b28d61d73b07c (table test-table [id=7c7b95a477ee42f684634af8845de140]) successfully deleted
W20260812 02:37:12.568049 5983 tablet.cc:2369] T c094f84039fc44a78f11a933ba7be6be P ec5174c38bbe431bb178024d344f137a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:37:12.575834 5742 tablet.cc:2369] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:37:12.583627 6194 tablet.cc:2369] T c094f84039fc44a78f11a933ba7be6be P 76c5ade3cbce49ec9e84c3d8dab52867: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:37:12.691180 6243 raft_consensus.cc:482] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:37:12.691346 6243 raft_consensus.cc:504] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.692703 6243 leader_election.cc:288] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec5174c38bbe431bb178024d344f137a (127.3.127.130:36409), 76c5ade3cbce49ec9e84c3d8dab52867 (127.3.127.131:39671)
I20260812 02:37:12.698693 6227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.693060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:55209 (local address 127.3.127.130:36409)
0812 02:37:12.693266 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:12.693271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:12.693375 (+ 104us) server_negotiation.cc:407] Connection header received
0812 02:37:12.693622 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:12.693626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:12.693681 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:12.693732 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:12.694291 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.695069 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.697442 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.697879 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.698005 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.698069 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:12.698421 (+ 352us) server_negotiation.cc:299] Negotiation successful
0812 02:37:12.698520 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 02:37:12.698860 6277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.692914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:36409 (local address 127.3.127.129:55209)
0812 02:37:12.693287 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:12.693302 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:12.693469 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:12.693878 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:12.693881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:12.693896 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:12.694143 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.694152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.695184 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.695188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.697228 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.697247 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.698023 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.698027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.698264 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.698298 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 02:37:12.698390 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 02:37:12.698483 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":46,"thread_start_us":88,"threads_started":1}
I20260812 02:37:12.699048 6225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.693501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:43695 (local address 127.3.127.131:39671)
0812 02:37:12.693634 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:12.693639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:12.693745 (+ 106us) server_negotiation.cc:407] Connection header received
0812 02:37:12.693851 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:12.693854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:12.693907 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:12.694006 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:12.694467 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.695025 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.697421 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.698085 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.698236 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.698290 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:12.698824 (+ 534us) server_negotiation.cc:299] Negotiation successful
0812 02:37:12.698925 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 02:37:12.699455 6278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.693357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:39671 (local address 127.3.127.129:43695)
0812 02:37:12.693659 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:12.693672 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:12.693763 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:12.694028 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:12.694031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:12.694046 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:12.694342 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.694353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.695144 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.695148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.697228 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.697247 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.698345 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.698350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.698589 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.698682 (+ 93us) client_negotiation.cc:769] Sending connection context
0812 02:37:12.699208 (+ 526us) client_negotiation.cc:240] Negotiation successful
0812 02:37:12.699296 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":73,"thread_start_us":79,"threads_started":1}
I20260812 02:37:12.699918 5909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c094f84039fc44a78f11a933ba7be6be" candidate_uuid: "220d893d0a444e069882aa6f3a1269af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec5174c38bbe431bb178024d344f137a" is_pre_election: true
I20260812 02:37:12.700297 5909 raft_consensus.cc:2454] T c094f84039fc44a78f11a933ba7be6be P ec5174c38bbe431bb178024d344f137a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 220d893d0a444e069882aa6f3a1269af in term 0.
I20260812 02:37:12.700292 6126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c094f84039fc44a78f11a933ba7be6be" candidate_uuid: "220d893d0a444e069882aa6f3a1269af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" is_pre_election: true
I20260812 02:37:12.700446 6126 raft_consensus.cc:2454] T c094f84039fc44a78f11a933ba7be6be P 76c5ade3cbce49ec9e84c3d8dab52867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 220d893d0a444e069882aa6f3a1269af in term 0.
I20260812 02:37:12.700827 5559 leader_election.cc:302] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 220d893d0a444e069882aa6f3a1269af, ec5174c38bbe431bb178024d344f137a; no voters:
I20260812 02:37:12.701133 6243 raft_consensus.cc:2781] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:37:12.701226 6243 raft_consensus.cc:482] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:37:12.701265 6243 raft_consensus.cc:3037] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:12.703140 6243 raft_consensus.cc:504] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.703608 6243 leader_election.cc:288] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af [CANDIDATE]: Term 1 election: Requested vote from peers ec5174c38bbe431bb178024d344f137a (127.3.127.130:36409), 76c5ade3cbce49ec9e84c3d8dab52867 (127.3.127.131:39671)
I20260812 02:37:12.704104 6126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c094f84039fc44a78f11a933ba7be6be" candidate_uuid: "220d893d0a444e069882aa6f3a1269af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c5ade3cbce49ec9e84c3d8dab52867"
I20260812 02:37:12.704125 5909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c094f84039fc44a78f11a933ba7be6be" candidate_uuid: "220d893d0a444e069882aa6f3a1269af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec5174c38bbe431bb178024d344f137a"
I20260812 02:37:12.704267 6126 raft_consensus.cc:3037] T c094f84039fc44a78f11a933ba7be6be P 76c5ade3cbce49ec9e84c3d8dab52867 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:12.704267 5909 raft_consensus.cc:3037] T c094f84039fc44a78f11a933ba7be6be P ec5174c38bbe431bb178024d344f137a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:12.705602 6126 raft_consensus.cc:2454] T c094f84039fc44a78f11a933ba7be6be P 76c5ade3cbce49ec9e84c3d8dab52867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 220d893d0a444e069882aa6f3a1269af in term 1.
I20260812 02:37:12.705958 5559 leader_election.cc:302] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 220d893d0a444e069882aa6f3a1269af, 76c5ade3cbce49ec9e84c3d8dab52867; no voters:
I20260812 02:37:12.706176 6243 raft_consensus.cc:2781] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:12.706419 6243 raft_consensus.cc:686] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af [term 1 LEADER]: Becoming Leader. State: Replica: 220d893d0a444e069882aa6f3a1269af, State: Running, Role: LEADER
I20260812 02:37:12.706514 5909 raft_consensus.cc:2454] T c094f84039fc44a78f11a933ba7be6be P ec5174c38bbe431bb178024d344f137a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 220d893d0a444e069882aa6f3a1269af in term 1.
I20260812 02:37:12.706938 6243 consensus_queue.cc:237] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.709965 5413 catalog_manager.cc:5495] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af reported cstate change: term changed from 0 to 1, leader changed from <none> to 220d893d0a444e069882aa6f3a1269af (127.3.127.129). New cstate: current_term: 1 leader_uuid: "220d893d0a444e069882aa6f3a1269af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } health_report { overall_health: UNKNOWN } } }
Aug 12 02:37:12 dist-test-slave-rmhg krb5kdc[5342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502231, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:12.781144 6290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.771401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41453 (local address 127.0.0.1:33654)
0812 02:37:12.771897 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:12.771911 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:12.772035 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:12.772282 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:12.772284 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:12.772823 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:12.772958 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.772964 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.773838 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.773841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.774420 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:12.774429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.774937 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.774940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:12.775015 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.775749 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:12.775763 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:12.777379 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:12.779228 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:12.779241 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:12.779248 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:12.779584 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:12.779982 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:12.779985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:12.779987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:12.780087 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:12.780538 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:12.780544 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:12.780679 (+ 135us) client_negotiation.cc:769] Sending connection context
0812 02:37:12.780792 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 02:37:12.780954 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":144,"threads_started":1}
I20260812 02:37:12.781183 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:12.771456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33654 (local address 127.3.127.190:41453)
0812 02:37:12.771598 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:12.771602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:12.771993 (+ 391us) server_negotiation.cc:407] Connection header received
0812 02:37:12.772144 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:12.772146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:12.772191 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:12.772251 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:12.773112 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.773692 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.774580 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:12.774801 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:12.774884 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:12.777600 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:12.777603 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:12.777605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:12.777679 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:12.779088 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:12.779738 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:12.779742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:12.779743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:12.779850 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:12.780170 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:12.780172 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:12.780173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:12.780373 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:12.780449 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:12.780846 (+ 397us) server_negotiation.cc:299] Negotiation successful
0812 02:37:12.781033 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":52}
I20260812 02:37:12.786223 5413 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:33654:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 02:37:12.787137 5413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:37:12.792703 5634 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb36662ec144f1396c263f5a163863a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1db3a965b97148bd91d2ad5875c4854d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:37:12.792821 5877 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb36662ec144f1396c263f5a163863a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1db3a965b97148bd91d2ad5875c4854d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:37:12.792881 6097 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb36662ec144f1396c263f5a163863a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1db3a965b97148bd91d2ad5875c4854d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:37:12.793303 5877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb36662ec144f1396c263f5a163863a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:12.793298 6097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb36662ec144f1396c263f5a163863a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:12.793300 5634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb36662ec144f1396c263f5a163863a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:12.796712 6234 tablet_bootstrap.cc:492] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867: Bootstrap starting.
I20260812 02:37:12.796712 6235 tablet_bootstrap.cc:492] T 7eb36662ec144f1396c263f5a163863a P ec5174c38bbe431bb178024d344f137a: Bootstrap starting.
I20260812 02:37:12.797906 6235 tablet_bootstrap.cc:654] T 7eb36662ec144f1396c263f5a163863a P ec5174c38bbe431bb178024d344f137a: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:12.798622 6234 tablet_bootstrap.cc:654] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:12.799393 6235 tablet_bootstrap.cc:492] T 7eb36662ec144f1396c263f5a163863a P ec5174c38bbe431bb178024d344f137a: No bootstrap required, opened a new log
I20260812 02:37:12.799477 6235 ts_tablet_manager.cc:1397] T 7eb36662ec144f1396c263f5a163863a P ec5174c38bbe431bb178024d344f137a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260812 02:37:12.799988 6230 tablet_bootstrap.cc:492] T 7eb36662ec144f1396c263f5a163863a P 220d893d0a444e069882aa6f3a1269af: Bootstrap starting.
I20260812 02:37:12.799921 6235 raft_consensus.cc:348] T 7eb36662ec144f1396c263f5a163863a P ec5174c38bbe431bb178024d344f137a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.800042 6235 raft_consensus.cc:374] T 7eb36662ec144f1396c263f5a163863a P ec5174c38bbe431bb178024d344f137a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:12.800099 6235 raft_consensus.cc:729] T 7eb36662ec144f1396c263f5a163863a P ec5174c38bbe431bb178024d344f137a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec5174c38bbe431bb178024d344f137a, State: Initialized, Role: FOLLOWER
I20260812 02:37:12.800276 6235 consensus_queue.cc:260] T 7eb36662ec144f1396c263f5a163863a P ec5174c38bbe431bb178024d344f137a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.800582 6247 tablet_replica.cc:439] T 7eb36662ec144f1396c263f5a163863a P ec5174c38bbe431bb178024d344f137a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } } }
I20260812 02:37:12.800760 6247 tablet_replica.cc:442] T 7eb36662ec144f1396c263f5a163863a P ec5174c38bbe431bb178024d344f137a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:37:12.801036 6235 ts_tablet_manager.cc:1428] T 7eb36662ec144f1396c263f5a163863a P ec5174c38bbe431bb178024d344f137a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260812 02:37:12.801208 6234 tablet_bootstrap.cc:492] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867: No bootstrap required, opened a new log
I20260812 02:37:12.801357 6234 ts_tablet_manager.cc:1397] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260812 02:37:12.801839 6230 tablet_bootstrap.cc:654] T 7eb36662ec144f1396c263f5a163863a P 220d893d0a444e069882aa6f3a1269af: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:12.802405 6234 raft_consensus.cc:348] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.802685 6234 raft_consensus.cc:374] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:12.802739 6234 raft_consensus.cc:729] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c5ade3cbce49ec9e84c3d8dab52867, State: Initialized, Role: FOLLOWER
I20260812 02:37:12.802884 6234 consensus_queue.cc:260] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.803207 6245 tablet_replica.cc:439] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } } }
I20260812 02:37:12.803483 6245 tablet_replica.cc:442] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:37:12.803752 6234 ts_tablet_manager.cc:1428] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 02:37:12.805265 6230 tablet_bootstrap.cc:492] T 7eb36662ec144f1396c263f5a163863a P 220d893d0a444e069882aa6f3a1269af: No bootstrap required, opened a new log
I20260812 02:37:12.805377 6230 ts_tablet_manager.cc:1397] T 7eb36662ec144f1396c263f5a163863a P 220d893d0a444e069882aa6f3a1269af: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260812 02:37:12.806138 6230 raft_consensus.cc:348] T 7eb36662ec144f1396c263f5a163863a P 220d893d0a444e069882aa6f3a1269af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.806241 6230 raft_consensus.cc:374] T 7eb36662ec144f1396c263f5a163863a P 220d893d0a444e069882aa6f3a1269af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:12.806262 6230 raft_consensus.cc:729] T 7eb36662ec144f1396c263f5a163863a P 220d893d0a444e069882aa6f3a1269af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220d893d0a444e069882aa6f3a1269af, State: Initialized, Role: FOLLOWER
I20260812 02:37:12.806375 6230 consensus_queue.cc:260] T 7eb36662ec144f1396c263f5a163863a P 220d893d0a444e069882aa6f3a1269af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:12.806643 6230 ts_tablet_manager.cc:1428] T 7eb36662ec144f1396c263f5a163863a P 220d893d0a444e069882aa6f3a1269af: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 02:37:12.806615 6243 tablet_replica.cc:439] T 7eb36662ec144f1396c263f5a163863a P 220d893d0a444e069882aa6f3a1269af: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } } }
I20260812 02:37:12.806751 6243 tablet_replica.cc:442] T 7eb36662ec144f1396c263f5a163863a P 220d893d0a444e069882aa6f3a1269af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:37:13.094535 6243 consensus_queue.cc:1035] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:13.099007 6243 consensus_queue.cc:1035] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:13.136122 6245 raft_consensus.cc:482] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:37:13.136296 6245 raft_consensus.cc:504] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:13.137579 6245 leader_election.cc:288] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 220d893d0a444e069882aa6f3a1269af (127.3.127.129:46309), ec5174c38bbe431bb178024d344f137a (127.3.127.130:36409)
I20260812 02:37:13.143288 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:13.137976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:58471 (local address 127.3.127.129:46309)
0812 02:37:13.138306 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:13.138321 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:13.138384 (+ 63us) server_negotiation.cc:407] Connection header received
0812 02:37:13.138549 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:13.138553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:13.138604 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:13.138695 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:13.139246 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.140021 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.141898 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.142640 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.142753 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:13.142802 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:13.143050 (+ 248us) server_negotiation.cc:299] Negotiation successful
0812 02:37:13.143134 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":88,"threads_started":1}
I20260812 02:37:13.143280 6227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:13.137957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:43639 (local address 127.3.127.130:36409)
0812 02:37:13.138114 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:13.138119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:13.138386 (+ 267us) server_negotiation.cc:407] Connection header received
0812 02:37:13.138549 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:13.138553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:13.138604 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:13.138694 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:13.139167 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.140016 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.141888 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.142521 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.142661 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:13.142755 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:13.143059 (+ 304us) server_negotiation.cc:299] Negotiation successful
0812 02:37:13.143135 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 02:37:13.143417 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:13.137848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:36409 (local address 127.3.127.131:43639)
0812 02:37:13.138273 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:13.138288 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:13.138444 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:13.138750 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:13.138754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:13.138785 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:13.139007 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:13.139016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.140140 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.140143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:13.141753 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:13.141768 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.142671 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.142674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:13.142884 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:13.142952 (+ 68us) client_negotiation.cc:769] Sending connection context
0812 02:37:13.143022 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 02:37:13.143164 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":187,"spinlock_wait_cycles":21248,"thread_start_us":101,"threads_started":1}
I20260812 02:37:13.143554 6308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:13.137848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:46309 (local address 127.3.127.131:58471)
0812 02:37:13.138312 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:13.138323 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:13.138443 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:13.138750 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:13.138754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:13.138785 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:13.139111 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:13.139121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.140132 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.140135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:13.141754 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:13.141767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.142750 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.142754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:13.142911 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:13.142944 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 02:37:13.143026 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 02:37:13.143164 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":345,"spinlock_wait_cycles":29824,"thread_start_us":91,"threads_started":1}
I20260812 02:37:13.143949 5909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb36662ec144f1396c263f5a163863a" candidate_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec5174c38bbe431bb178024d344f137a" is_pre_election: true
I20260812 02:37:13.144114 5909 raft_consensus.cc:2454] T 7eb36662ec144f1396c263f5a163863a P ec5174c38bbe431bb178024d344f137a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c5ade3cbce49ec9e84c3d8dab52867 in term 0.
I20260812 02:37:13.144263 5665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb36662ec144f1396c263f5a163863a" candidate_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220d893d0a444e069882aa6f3a1269af" is_pre_election: true
I20260812 02:37:13.144415 5665 raft_consensus.cc:2454] T 7eb36662ec144f1396c263f5a163863a P 220d893d0a444e069882aa6f3a1269af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c5ade3cbce49ec9e84c3d8dab52867 in term 0.
I20260812 02:37:13.144543 6018 leader_election.cc:302] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c5ade3cbce49ec9e84c3d8dab52867, ec5174c38bbe431bb178024d344f137a; no voters:
I20260812 02:37:13.144860 6245 raft_consensus.cc:2781] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:37:13.144924 6245 raft_consensus.cc:482] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:37:13.144963 6245 raft_consensus.cc:3037] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:13.146440 6245 raft_consensus.cc:504] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:13.146792 6245 leader_election.cc:288] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867 [CANDIDATE]: Term 1 election: Requested vote from peers 220d893d0a444e069882aa6f3a1269af (127.3.127.129:46309), ec5174c38bbe431bb178024d344f137a (127.3.127.130:36409)
I20260812 02:37:13.147162 5665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb36662ec144f1396c263f5a163863a" candidate_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220d893d0a444e069882aa6f3a1269af"
I20260812 02:37:13.147187 5909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb36662ec144f1396c263f5a163863a" candidate_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec5174c38bbe431bb178024d344f137a"
I20260812 02:37:13.147274 5665 raft_consensus.cc:3037] T 7eb36662ec144f1396c263f5a163863a P 220d893d0a444e069882aa6f3a1269af [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:13.147315 5909 raft_consensus.cc:3037] T 7eb36662ec144f1396c263f5a163863a P ec5174c38bbe431bb178024d344f137a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:13.148238 5665 raft_consensus.cc:2454] T 7eb36662ec144f1396c263f5a163863a P 220d893d0a444e069882aa6f3a1269af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c5ade3cbce49ec9e84c3d8dab52867 in term 1.
I20260812 02:37:13.148366 5909 raft_consensus.cc:2454] T 7eb36662ec144f1396c263f5a163863a P ec5174c38bbe431bb178024d344f137a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c5ade3cbce49ec9e84c3d8dab52867 in term 1.
I20260812 02:37:13.148524 6017 leader_election.cc:302] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 220d893d0a444e069882aa6f3a1269af, 76c5ade3cbce49ec9e84c3d8dab52867; no voters:
I20260812 02:37:13.148702 6245 raft_consensus.cc:2781] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:13.148913 6245 raft_consensus.cc:686] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867 [term 1 LEADER]: Becoming Leader. State: Replica: 76c5ade3cbce49ec9e84c3d8dab52867, State: Running, Role: LEADER
I20260812 02:37:13.149214 6245 consensus_queue.cc:237] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } }
I20260812 02:37:13.150635 6309 tablet_replica.cc:439] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867: TxnStatusTablet state changed. Reason: New leader 76c5ade3cbce49ec9e84c3d8dab52867. Latest consensus state: current_term: 1 leader_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } } }
I20260812 02:37:13.150802 6309 tablet_replica.cc:442] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:37:13.151276 6313 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 02:37:13.151510 6313 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 02:37:13.151871 5413 catalog_manager.cc:5495] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c5ade3cbce49ec9e84c3d8dab52867 (127.3.127.131). New cstate: current_term: 1 leader_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } health_report { overall_health: HEALTHY } } }
I20260812 02:37:13.173237 6290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:13.169131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:39671 (local address 127.0.0.1:57604)
0812 02:37:13.169347 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:13.169364 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:13.169492 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:13.169846 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:13.169849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:13.169858 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:37:13.169970 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:13.169975 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.170847 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.170851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:13.171515 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:13.171525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.172017 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.172021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:13.172109 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:13.172165 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:37:13.172873 (+ 708us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:37:13.172892 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 02:37:13.172963 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 02:37:13.173070 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260812 02:37:13.173370 6225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:13.169299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57604 (local address 127.3.127.131:39671)
0812 02:37:13.169429 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:13.169433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:13.169449 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:37:13.169652 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:13.169655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:13.169721 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:13.169814 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:13.170085 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.170734 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.171637 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.171905 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.171999 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:13.172317 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:13.172777 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:13.172886 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:13.172982 (+ 96us) server_negotiation.cc:299] Negotiation successful
0812 02:37:13.173048 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 02:37:13.177485 5665 raft_consensus.cc:1261] T 7eb36662ec144f1396c263f5a163863a P 220d893d0a444e069882aa6f3a1269af [term 1 FOLLOWER]: Refusing update from remote peer 76c5ade3cbce49ec9e84c3d8dab52867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:13.177615 5909 raft_consensus.cc:1261] T 7eb36662ec144f1396c263f5a163863a P ec5174c38bbe431bb178024d344f137a [term 1 FOLLOWER]: Refusing update from remote peer 76c5ade3cbce49ec9e84c3d8dab52867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:13.178186 6245 consensus_queue.cc:1035] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:13.178396 6309 consensus_queue.cc:1035] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:13.180181 6243 tablet_replica.cc:439] T 7eb36662ec144f1396c263f5a163863a P 220d893d0a444e069882aa6f3a1269af: TxnStatusTablet state changed. Reason: New leader 76c5ade3cbce49ec9e84c3d8dab52867. Latest consensus state: current_term: 1 leader_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } } }
I20260812 02:37:13.180330 6243 tablet_replica.cc:442] T 7eb36662ec144f1396c263f5a163863a P 220d893d0a444e069882aa6f3a1269af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:37:13.181355 6247 tablet_replica.cc:439] T 7eb36662ec144f1396c263f5a163863a P ec5174c38bbe431bb178024d344f137a: TxnStatusTablet state changed. Reason: New leader 76c5ade3cbce49ec9e84c3d8dab52867. Latest consensus state: current_term: 1 leader_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } } }
I20260812 02:37:13.181492 6247 tablet_replica.cc:442] T 7eb36662ec144f1396c263f5a163863a P ec5174c38bbe431bb178024d344f137a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:37:13.183669 6245 tablet_replica.cc:439] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } } }
I20260812 02:37:13.183818 6245 tablet_replica.cc:442] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:37:13.183948 6245 tablet_replica.cc:439] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } } }
I20260812 02:37:13.184036 6245 tablet_replica.cc:442] T 7eb36662ec144f1396c263f5a163863a P 76c5ade3cbce49ec9e84c3d8dab52867: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:37:13.184224 6300 mvcc.cc:205] Tried to move back new op lower bound from 7317513147088867328 to 7317513146981896192. Current Snapshot: MvccSnapshot[applied={T|T < 7317513147088867328}]
I20260812 02:37:13.184502 6247 tablet_replica.cc:439] T 7eb36662ec144f1396c263f5a163863a P ec5174c38bbe431bb178024d344f137a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } } }
I20260812 02:37:13.185086 6243 tablet_replica.cc:439] T 7eb36662ec144f1396c263f5a163863a P 220d893d0a444e069882aa6f3a1269af: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220d893d0a444e069882aa6f3a1269af" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46309 } } peers { permanent_uuid: "ec5174c38bbe431bb178024d344f137a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36409 } } peers { permanent_uuid: "76c5ade3cbce49ec9e84c3d8dab52867" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39671 } } }
I20260812 02:37:13.185240 6243 tablet_replica.cc:442] T 7eb36662ec144f1396c263f5a163863a P 220d893d0a444e069882aa6f3a1269af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:37:13.185530 6247 tablet_replica.cc:442] T 7eb36662ec144f1396c263f5a163863a P ec5174c38bbe431bb178024d344f137a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:37:13.196906 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:13.193188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39386 (local address 127.3.127.129:46309)
0812 02:37:13.193316 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:13.193320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:13.193331 (+ 11us) server_negotiation.cc:407] Connection header received
0812 02:37:13.193411 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:13.193413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:13.193470 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:13.193565 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:13.193834 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.194352 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.195384 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.195618 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.195711 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:13.196005 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:13.196490 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:13.196620 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:13.196683 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 02:37:13.196755 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 02:37:13.208268 6225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:13.205067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:34391 (local address 127.3.127.131:39671)
0812 02:37:13.205271 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:13.205274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:13.205286 (+ 12us) server_negotiation.cc:407] Connection header received
0812 02:37:13.205329 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:13.205331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:13.205373 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:13.205438 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:13.205710 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.206300 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.207199 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.207413 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.207500 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:13.207811 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:13.207895 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:13.207954 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:13.208053 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 02:37:13.208115 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260812 02:37:13.208328 6331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:13.204676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:39671 (local address 127.3.127.129:34391)
0812 02:37:13.205033 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:13.205046 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:13.205141 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:13.205459 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:13.205461 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:13.205469 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:37:13.205568 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:13.205573 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.206412 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.206415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:13.207077 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:13.207087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.207519 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.207522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:13.207629 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:13.207680 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:37:13.207971 (+ 291us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:37:13.207978 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 02:37:13.208035 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 02:37:13.208128 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":124,"threads_started":1}
I20260812 02:37:13.240111 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:13.235473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:56449 (local address 127.3.127.129:46309)
0812 02:37:13.235615 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:13.235620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:13.236101 (+ 481us) server_negotiation.cc:407] Connection header received
0812 02:37:13.236224 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:13.236228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:13.236289 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:13.236360 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:13.236940 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.237638 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.238729 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.238959 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.239053 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:13.239572 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:13.239659 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:13.239729 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:13.239895 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 02:37:13.239969 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 02:37:13.240190 6337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:13.235446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:46309 (local address 127.3.127.131:56449)
0812 02:37:13.236004 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:13.236018 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:13.236121 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:13.236495 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:13.236498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:13.236507 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:37:13.236822 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:13.236831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.237780 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.237784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:13.238595 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:13.238607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.239217 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.239222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:13.239344 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:13.239403 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:37:13.239775 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:37:13.239782 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 02:37:13.239849 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 02:37:13.239941 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":168,"threads_started":1}
I20260812 02:37:13.271507 5413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:13.271740 5413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:13.274438 5413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34ce3a8807e3477999428a2e306c9b25: Sending DeleteTablet for 3 replicas of tablet c094f84039fc44a78f11a933ba7be6be
I20260812 02:37:13.275331 5877 tablet_service.cc:1514] Processing DeleteTablet for tablet c094f84039fc44a78f11a933ba7be6be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:13 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:37828
I20260812 02:37:13.276033 6340 tablet_replica.cc:330] T c094f84039fc44a78f11a933ba7be6be P ec5174c38bbe431bb178024d344f137a: stopping tablet replica
I20260812 02:37:13.276315 6340 raft_consensus.cc:2229] T c094f84039fc44a78f11a933ba7be6be P ec5174c38bbe431bb178024d344f137a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:13.276528 6340 raft_consensus.cc:2258] T c094f84039fc44a78f11a933ba7be6be P ec5174c38bbe431bb178024d344f137a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:13.276894 5634 tablet_service.cc:1514] Processing DeleteTablet for tablet c094f84039fc44a78f11a933ba7be6be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:13 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:39382
I20260812 02:37:13.277416 6340 ts_tablet_manager.cc:1905] T c094f84039fc44a78f11a933ba7be6be P ec5174c38bbe431bb178024d344f137a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:13.277910 6097 tablet_service.cc:1514] Processing DeleteTablet for tablet c094f84039fc44a78f11a933ba7be6be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:13 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:57596
I20260812 02:37:13.278009 6342 tablet_replica.cc:330] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af: stopping tablet replica
I20260812 02:37:13.278177 6342 raft_consensus.cc:2229] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:37:13.278544 6342 raft_consensus.cc:2258] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:13.279417 6342 ts_tablet_manager.cc:1905] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:13.282446 6340 ts_tablet_manager.cc:1918] T c094f84039fc44a78f11a933ba7be6be P ec5174c38bbe431bb178024d344f137a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:37:13.282645 6340 log.cc:1196] T c094f84039fc44a78f11a933ba7be6be P ec5174c38bbe431bb178024d344f137a: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/c094f84039fc44a78f11a933ba7be6be
I20260812 02:37:13.283028 6340 ts_tablet_manager.cc:1939] T c094f84039fc44a78f11a933ba7be6be P ec5174c38bbe431bb178024d344f137a: Deleting consensus metadata
I20260812 02:37:13.284116 5384 catalog_manager.cc:4841] TS ec5174c38bbe431bb178024d344f137a (127.3.127.130:36409): tablet c094f84039fc44a78f11a933ba7be6be (table test-table [id=7192696e8bcb40fcbaaed4588bf27ace]) successfully deleted
I20260812 02:37:13.284237 6342 ts_tablet_manager.cc:1918] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:37:13.284327 6342 log.cc:1196] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-0/wal/wals/c094f84039fc44a78f11a933ba7be6be
I20260812 02:37:13.284621 6342 ts_tablet_manager.cc:1939] T c094f84039fc44a78f11a933ba7be6be P 220d893d0a444e069882aa6f3a1269af: Deleting consensus metadata
I20260812 02:37:13.284713 6345 tablet_replica.cc:330] T c094f84039fc44a78f11a933ba7be6be P 76c5ade3cbce49ec9e84c3d8dab52867: stopping tablet replica
I20260812 02:37:13.284929 6345 raft_consensus.cc:2229] T c094f84039fc44a78f11a933ba7be6be P 76c5ade3cbce49ec9e84c3d8dab52867 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:13.285261 6345 raft_consensus.cc:2258] T c094f84039fc44a78f11a933ba7be6be P 76c5ade3cbce49ec9e84c3d8dab52867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:13.285683 5382 catalog_manager.cc:4841] TS 220d893d0a444e069882aa6f3a1269af (127.3.127.129:46309): tablet c094f84039fc44a78f11a933ba7be6be (table test-table [id=7192696e8bcb40fcbaaed4588bf27ace]) successfully deleted
I20260812 02:37:13.286132 6345 ts_tablet_manager.cc:1905] T c094f84039fc44a78f11a933ba7be6be P 76c5ade3cbce49ec9e84c3d8dab52867: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:13.290068 6345 ts_tablet_manager.cc:1918] T c094f84039fc44a78f11a933ba7be6be P 76c5ade3cbce49ec9e84c3d8dab52867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:37:13.290278 6345 log.cc:1196] T c094f84039fc44a78f11a933ba7be6be P 76c5ade3cbce49ec9e84c3d8dab52867: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/c094f84039fc44a78f11a933ba7be6be
I20260812 02:37:13.290858 6345 ts_tablet_manager.cc:1939] T c094f84039fc44a78f11a933ba7be6be P 76c5ade3cbce49ec9e84c3d8dab52867: Deleting consensus metadata
I20260812 02:37:13.291884 5384 catalog_manager.cc:4841] TS 76c5ade3cbce49ec9e84c3d8dab52867 (127.3.127.131:39671): tablet c094f84039fc44a78f11a933ba7be6be (table test-table [id=7192696e8bcb40fcbaaed4588bf27ace]) successfully deleted
Aug 12 02:37:13 dist-test-slave-rmhg krb5kdc[5342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502232, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.127.129@KRBTEST.COM
I20260812 02:37:13.305635 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:13.278189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39394 (local address 127.3.127.129:46309)
0812 02:37:13.278352 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:13.278357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:13.290036 (+ 11679us) server_negotiation.cc:407] Connection header received
0812 02:37:13.290268 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:13.290272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:13.290326 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:13.290380 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:13.291421 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.292245 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.294068 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.294446 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.294617 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:13.300015 (+ 5398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:13.300062 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:13.300093 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:13.300139 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:13.302813 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:13.303575 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:13.303582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:13.303585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:13.303698 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:13.304086 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:13.304090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:13.304092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:13.304532 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:13.304788 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:13.305281 (+ 493us) server_negotiation.cc:299] Negotiation successful
0812 02:37:13.305441 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260812 02:37:13.306280 5733 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39394
I20260812 02:37:13.319198 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:13.308491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33656 (local address 127.3.127.190:41453)
0812 02:37:13.309132 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:13.309139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:13.309197 (+ 58us) server_negotiation.cc:407] Connection header received
0812 02:37:13.309345 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:13.309348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:13.309414 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:13.309516 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:13.310431 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.311259 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.312543 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:13.312941 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:13.313065 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:13.315087 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:13.315091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:13.315094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:13.315133 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:13.317089 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:13.317713 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:13.317718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:13.317720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:13.317832 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:13.318139 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:13.318143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:13.318145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:13.318412 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:13.318559 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:13.318845 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 02:37:13.318997 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":81,"threads_started":1}
W20260812 02:37:13.319754 5413 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33656
I20260812 02:37:13.321676 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 5527
I20260812 02:37:13.342844 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 5760
I20260812 02:37:13.363161 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 5994
I20260812 02:37:13.378969 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 5361
2026-08-12T02:37:13Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3391 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:37:13 dist-test-slave-rmhg krb5kdc[6354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:37:13 dist-test-slave-rmhg krb5kdc[6354](info): set up 2 sockets
Aug 12 02:37:13 dist-test-slave-rmhg krb5kdc[6354](info): commencing operation
krb5kdc: starting...
W20260812 02:37:14.563740 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:37:14 dist-test-slave-rmhg krb5kdc[6354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:37:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:37:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:37:14.694293 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:33073
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:34211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:33073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:14.812868 6375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:14.813184 6375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:14.813248 6375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:14.817317 6375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:14.817384 6375 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 02:37:14.817406 6375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:14.817430 6375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:14.817452 6375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:14.817477 6375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:37:14.822893 6375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:34211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:33073
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:33073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:14.824067 6375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:14.825182 6375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:14.834084 6385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:14.834086 6388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:14.834091 6386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:14.834622 6375 server_base.cc:1034] running on GCE node
I20260812 02:37:14.835243 6375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:14.836520 6375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:14.837755 6375 hybrid_clock.cc:648] HybridClock initialized: now 1786502234837736 us; error 39 us; skew 500 ppm
Aug 12 02:37:14 dist-test-slave-rmhg krb5kdc[6354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502234, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:14.841243 6375 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:37:14.842355 6375 webserver.cc:460] Webserver started at http://127.3.127.190:43197/ using document root <none> and password file <none>
I20260812 02:37:14.843005 6375 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:14.843078 6375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:14.843297 6375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:14.845736 6375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "dee11b974a3d4738b8f9a6170354282c"
format_stamp: "Formatted at 2026-08-12 02:37:14 on dist-test-slave-rmhg"
server_key: "3a0ede130b1160ced341291d6653dab5"
server_key_iv: "f00160d74418ae5d4c3dc48dc1a08acc"
server_key_version: "encryptionkey@0"
I20260812 02:37:14.846359 6375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "dee11b974a3d4738b8f9a6170354282c"
format_stamp: "Formatted at 2026-08-12 02:37:14 on dist-test-slave-rmhg"
server_key: "3a0ede130b1160ced341291d6653dab5"
server_key_iv: "f00160d74418ae5d4c3dc48dc1a08acc"
server_key_version: "encryptionkey@0"
I20260812 02:37:14.850653 6375 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:37:14.853446 6398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:14.854920 6375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:14.855079 6375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "dee11b974a3d4738b8f9a6170354282c"
format_stamp: "Formatted at 2026-08-12 02:37:14 on dist-test-slave-rmhg"
server_key: "3a0ede130b1160ced341291d6653dab5"
server_key_iv: "f00160d74418ae5d4c3dc48dc1a08acc"
server_key_version: "encryptionkey@0"
I20260812 02:37:14.855211 6375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:14.867203 6375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:14.870733 6375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:14.870929 6375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:14.879580 6375 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:33073
I20260812 02:37:14.879591 6511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:33073 every 8 connection(s)
I20260812 02:37:14.880636 6375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:37:14.882160 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 6375
I20260812 02:37:14.882282 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:37:14.882578 3582 external_mini_cluster.cc:1428] Setting key 1024f439213b4ae4f96b03374c79f09f
I20260812 02:37:14.884128 6516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:14.890228 6516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c: Bootstrap starting.
Aug 12 02:37:14 dist-test-slave-rmhg krb5kdc[6354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:14.892964 6516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:14.893761 6516 log.cc:824] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:14.896270 6516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c: No bootstrap required, opened a new log
I20260812 02:37:14.898515 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:14.884167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46816 (local address 127.3.127.190:33073)
0812 02:37:14.884643 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:14.884661 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:14.884700 (+ 39us) server_negotiation.cc:407] Connection header received
0812 02:37:14.885472 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:14.885493 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:14.885875 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:14.886222 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:14.887074 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:14.888120 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:14.889020 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:14.889602 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:14.889774 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:14.892576 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:14.892582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:14.892598 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:14.892634 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:14.895248 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:14.895796 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:14.895800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:14.895806 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:14.895971 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:14.896338 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:14.896341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:14.896343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:14.896828 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:14.897012 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:14.897376 (+ 364us) server_negotiation.cc:299] Negotiation successful
0812 02:37:14.897697 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":150,"threads_started":1}
I20260812 02:37:14.899705 6516 raft_consensus.cc:348] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee11b974a3d4738b8f9a6170354282c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33073 } }
I20260812 02:37:14.900010 6516 raft_consensus.cc:374] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:14.900071 6516 raft_consensus.cc:729] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee11b974a3d4738b8f9a6170354282c, State: Initialized, Role: FOLLOWER
I20260812 02:37:14.900563 6516 consensus_queue.cc:260] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee11b974a3d4738b8f9a6170354282c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33073 } }
I20260812 02:37:14.900739 6516 raft_consensus.cc:388] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:37:14.900839 6516 raft_consensus.cc:482] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:37:14.900946 6516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:14.902002 6516 raft_consensus.cc:504] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee11b974a3d4738b8f9a6170354282c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33073 } }
I20260812 02:37:14.902343 6516 leader_election.cc:302] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dee11b974a3d4738b8f9a6170354282c; no voters:
I20260812 02:37:14.902630 6516 leader_election.cc:288] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:37:14.902753 6522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:14.902990 6522 raft_consensus.cc:686] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c [term 1 LEADER]: Becoming Leader. State: Replica: dee11b974a3d4738b8f9a6170354282c, State: Running, Role: LEADER
I20260812 02:37:14.903340 6522 consensus_queue.cc:237] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee11b974a3d4738b8f9a6170354282c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33073 } }
I20260812 02:37:14.903998 6516 sys_catalog.cc:564] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:37:14.905194 6523 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dee11b974a3d4738b8f9a6170354282c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee11b974a3d4738b8f9a6170354282c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33073 } } }
I20260812 02:37:14.905391 6523 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:14.905189 6524 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c [sys.catalog]: SysCatalogTable state changed. Reason: New leader dee11b974a3d4738b8f9a6170354282c. Latest consensus state: current_term: 1 leader_uuid: "dee11b974a3d4738b8f9a6170354282c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee11b974a3d4738b8f9a6170354282c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33073 } } }
I20260812 02:37:14.905599 6524 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:14.905845 6539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:37:14.909035 6539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:37:14.914906 6539 catalog_manager.cc:1347] Generated new cluster ID: c650d832215445c796e2e2cca90d7774
I20260812 02:37:14.915000 6539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:37:14.930080 6539 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:37:14.931207 6539 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:37:14.950912 6539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c: Generated new TSK 0
I20260812 02:37:14.951884 6539 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
I20260812 02:37:14.992853 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33073
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:34211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:15.116268 6560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:15.116575 6560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:15.116627 6560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:15.120874 6560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:15.120949 6560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:15.120995 6560 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 02:37:15.121023 6560 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:37:15.121074 6560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:37:15.126713 6560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:34211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.127.190:33073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:15.127931 6560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:15.129045 6560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:15.140405 6570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:15.140405 6576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:15.140484 6573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:15.140597 6560 server_base.cc:1034] running on GCE node
I20260812 02:37:15.141340 6560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:15.142097 6560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:15.143311 6560 hybrid_clock.cc:648] HybridClock initialized: now 1786502235143273 us; error 52 us; skew 500 ppm
Aug 12 02:37:15 dist-test-slave-rmhg krb5kdc[6354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502235, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:15.146585 6560 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:37:15.147616 6560 webserver.cc:460] Webserver started at http://127.3.127.129:37771/ using document root <none> and password file <none>
I20260812 02:37:15.148206 6560 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:15.148268 6560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:15.148453 6560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:15.150728 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "477261cc28a74a3cbfadc6e51e6a0bf5"
format_stamp: "Formatted at 2026-08-12 02:37:15 on dist-test-slave-rmhg"
server_key: "e3288f53a48bf609a32f13e15f25d4f6"
server_key_iv: "8a355b30fa277f85a625ddbe142e9c31"
server_key_version: "encryptionkey@0"
I20260812 02:37:15.151356 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "477261cc28a74a3cbfadc6e51e6a0bf5"
format_stamp: "Formatted at 2026-08-12 02:37:15 on dist-test-slave-rmhg"
server_key: "e3288f53a48bf609a32f13e15f25d4f6"
server_key_iv: "8a355b30fa277f85a625ddbe142e9c31"
server_key_version: "encryptionkey@0"
I20260812 02:37:15.155479 6560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:37:15.158157 6585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:15.159536 6560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:15.159669 6560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "477261cc28a74a3cbfadc6e51e6a0bf5"
format_stamp: "Formatted at 2026-08-12 02:37:15 on dist-test-slave-rmhg"
server_key: "e3288f53a48bf609a32f13e15f25d4f6"
server_key_iv: "8a355b30fa277f85a625ddbe142e9c31"
server_key_version: "encryptionkey@0"
I20260812 02:37:15.159803 6560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:15.193871 6560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:15.197234 6560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:15.197412 6560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:15.199004 6560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:15.199079 6560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:15.199138 6560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:15.199164 6560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:37:15 dist-test-slave-rmhg krb5kdc[6354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502235, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:15.209883 6560 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:33869
I20260812 02:37:15.209968 6791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:33869 every 8 connection(s)
I20260812 02:37:15.210857 6560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:37:15.211843 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 6560
I20260812 02:37:15.211952 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:37:15.212277 3582 external_mini_cluster.cc:1428] Setting key c902a5798ea1dc23890539cb750ffedc
I20260812 02:37:15.214831 6603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.200573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:33073 (local address 127.3.127.129:57717)
0812 02:37:15.201093 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:15.201140 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:15.202152 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:15.203038 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:15.203048 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:15.203500 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:15.204257 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.204280 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.205448 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.205452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.206624 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.206636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.207280 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.207284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.207472 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.208069 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:15.208089 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:15.209668 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:15.211617 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:15.211623 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:15.211638 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:15.211964 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:15.212314 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:15.212317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:15.212319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:15.212576 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:15.213110 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:15.213116 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:15.213414 (+ 298us) client_negotiation.cc:769] Sending connection context
0812 02:37:15.213658 (+ 244us) client_negotiation.cc:240] Negotiation successful
0812 02:37:15.214003 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":131,"threads_started":1}
I20260812 02:37:15.215533 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.199964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:57717 (local address 127.3.127.190:33073)
0812 02:37:15.200090 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:15.200093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:15.201286 (+ 1193us) server_negotiation.cc:407] Connection header received
0812 02:37:15.202455 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:15.202459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:15.202524 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:15.202583 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:15.204422 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.205317 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.206761 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.207106 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.207210 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.209861 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:15.209865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:15.209868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:15.209899 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:15.211482 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:15.212068 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:15.212072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:15.212074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:15.212179 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:15.212727 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:15.212730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:15.212732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:15.212977 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:15.213083 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:15.215245 (+ 2162us) server_negotiation.cc:299] Negotiation successful
0812 02:37:15.215391 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 02:37:15.216959 6793 heartbeater.cc:347] Connected to a master server at 127.3.127.190:33073
I20260812 02:37:15.217142 6793 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:15.217736 6793 heartbeater.cc:511] Master 127.3.127.190:33073 requested a full tablet report, sending...
I20260812 02:37:15.219343 6434 ts_manager.cc:194] Registered new tserver with Master: 477261cc28a74a3cbfadc6e51e6a0bf5 (127.3.127.129:33869)
I20260812 02:37:15.220702 6434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:57717
WARNING: no policy specified for kudu/127.3.127.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.130@KRBTEST.COM" created.
I20260812 02:37:15.228881 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.221448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:48641 (local address 127.3.127.190:33073)
0812 02:37:15.221604 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:15.221609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:15.221626 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:37:15.221667 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:15.221670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:15.221726 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:15.221815 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:15.222349 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.223182 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.224110 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.224416 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.224522 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.225687 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:15.225691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:15.225693 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:15.225724 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:15.227177 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:15.227643 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:15.227646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:15.227647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:15.227732 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:15.228007 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:15.228011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:15.228013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:15.228219 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:15.228309 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:15.228587 (+ 278us) server_negotiation.cc:299] Negotiation successful
0812 02:37:15.228724 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 02:37:15.228894 6807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.219596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:33073 (local address 127.3.127.129:48641)
0812 02:37:15.219946 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:15.219963 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:15.220065 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:15.221844 (+ 1779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:15.221847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:15.222079 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:15.222211 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.222219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.223337 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.223340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.223973 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.223984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.224549 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.224552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.224657 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.225171 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:15.225189 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:15.225542 (+ 353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:15.227295 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:15.227299 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:15.227301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:15.227513 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:15.227827 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:15.227831 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:15.227833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:15.227918 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:15.228317 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:15.228320 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:15.228421 (+ 101us) client_negotiation.cc:769] Sending connection context
0812 02:37:15.228511 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 02:37:15.228694 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
I20260812 02:37:15.244102 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33073
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:34211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:15.366236 6810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:15.366535 6810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:15.366585 6810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:15.370679 6810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:15.370765 6810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:15.370821 6810 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 02:37:15.370846 6810 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:37:15.370885 6810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:37:15.376533 6810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:34211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.127.190:33073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:15.377703 6810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:15.378795 6810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:15.388892 6823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:15.388891 6827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:15.389045 6810 server_base.cc:1034] running on GCE node
W20260812 02:37:15.389253 6824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:15.389693 6810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:15.390353 6810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:15.391520 6810 hybrid_clock.cc:648] HybridClock initialized: now 1786502235391485 us; error 49 us; skew 500 ppm
Aug 12 02:37:15 dist-test-slave-rmhg krb5kdc[6354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502235, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:15.394375 6810 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:37:15.395318 6810 webserver.cc:460] Webserver started at http://127.3.127.130:42765/ using document root <none> and password file <none>
I20260812 02:37:15.395917 6810 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:15.395992 6810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:15.396239 6810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:15.398546 6810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "8f35f8c70bc8444ebda49652410026fa"
format_stamp: "Formatted at 2026-08-12 02:37:15 on dist-test-slave-rmhg"
server_key: "c195daa2ed761cb9ab3961a6280f1c8a"
server_key_iv: "141ea74f39ca0921c4e02ae181900634"
server_key_version: "encryptionkey@0"
I20260812 02:37:15.399168 6810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "8f35f8c70bc8444ebda49652410026fa"
format_stamp: "Formatted at 2026-08-12 02:37:15 on dist-test-slave-rmhg"
server_key: "c195daa2ed761cb9ab3961a6280f1c8a"
server_key_iv: "141ea74f39ca0921c4e02ae181900634"
server_key_version: "encryptionkey@0"
I20260812 02:37:15.403334 6810 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:37:15.406293 6835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:15.407636 6810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:15.407785 6810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "8f35f8c70bc8444ebda49652410026fa"
format_stamp: "Formatted at 2026-08-12 02:37:15 on dist-test-slave-rmhg"
server_key: "c195daa2ed761cb9ab3961a6280f1c8a"
server_key_iv: "141ea74f39ca0921c4e02ae181900634"
server_key_version: "encryptionkey@0"
I20260812 02:37:15.407900 6810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:15.427625 6810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:15.430989 6810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:15.431205 6810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:15.432801 6810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:15.432857 6810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:15.432914 6810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:15.432951 6810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:37:15 dist-test-slave-rmhg krb5kdc[6354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502235, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:15.443895 6810 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:41201
I20260812 02:37:15.443989 7028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:41201 every 8 connection(s)
I20260812 02:37:15.445005 6810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:37:15.447167 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.433835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:38165 (local address 127.3.127.190:33073)
0812 02:37:15.433951 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:15.433954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:15.435134 (+ 1180us) server_negotiation.cc:407] Connection header received
0812 02:37:15.436216 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:15.436220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:15.436288 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:15.436390 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:15.437908 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.438456 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.440285 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.440622 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.440737 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.443622 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:15.443626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:15.443629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:15.443663 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:15.445104 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:15.445707 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:15.445710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:15.445712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:15.445813 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:15.446173 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:15.446175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:15.446176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:15.446366 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:15.446443 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:15.446914 (+ 471us) server_negotiation.cc:299] Negotiation successful
0812 02:37:15.447033 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 02:37:15.448038 6850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.434446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:33073 (local address 127.3.127.130:38165)
0812 02:37:15.434959 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:15.435022 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:15.435942 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:15.436643 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:15.436653 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:15.437048 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:15.437737 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.437756 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.438587 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.438591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.440131 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.440145 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.440781 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.440785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.440980 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.441641 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:15.441658 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:15.443396 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:15.445245 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:15.445251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:15.445268 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:15.445584 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:15.445916 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:15.445918 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:15.445920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:15.446081 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:15.446443 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:15.446448 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:15.446693 (+ 245us) client_negotiation.cc:769] Sending connection context
0812 02:37:15.446851 (+ 158us) client_negotiation.cc:240] Negotiation successful
0812 02:37:15.447153 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":132,"threads_started":1}
I20260812 02:37:15.449540 7029 heartbeater.cc:347] Connected to a master server at 127.3.127.190:33073
I20260812 02:37:15.449707 7029 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:15.450222 7029 heartbeater.cc:511] Master 127.3.127.190:33073 requested a full tablet report, sending...
I20260812 02:37:15.451400 6434 ts_manager.cc:194] Registered new tserver with Master: 8f35f8c70bc8444ebda49652410026fa (127.3.127.130:41201)
I20260812 02:37:15.452109 6434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:38165
I20260812 02:37:15.452239 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 6810
I20260812 02:37:15.452348 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:37:15.452620 3582 external_mini_cluster.cc:1428] Setting key ebbff088c75c369381134b8c022536a0
I20260812 02:37:15.459328 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.451857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:43975 (local address 127.3.127.190:33073)
0812 02:37:15.451992 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:15.451997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:15.452231 (+ 234us) server_negotiation.cc:407] Connection header received
0812 02:37:15.452400 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:15.452404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:15.452452 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:15.452527 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:15.453082 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.453812 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.454728 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.455010 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.455128 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.456261 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:15.456265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:15.456268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:15.456292 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:15.457648 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:15.458077 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:15.458080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:15.458081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:15.458160 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:15.458418 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:15.458421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:15.458423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:15.458667 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:15.458758 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:15.459035 (+ 277us) server_negotiation.cc:299] Negotiation successful
0812 02:37:15.459191 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":50}
I20260812 02:37:15.459338 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.451779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:33073 (local address 127.3.127.130:43975)
0812 02:37:15.452147 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:15.452161 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:15.452312 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:15.452550 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:15.452553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:15.452812 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:15.452953 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.452961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.453978 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.453981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.454605 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.454615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.455141 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.455144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.455223 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.455766 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:15.455780 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:15.456120 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:15.457758 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:15.457762 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:15.457764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:15.457967 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:15.458247 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:15.458250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:15.458252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:15.458338 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:15.458758 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:15.458761 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:15.458879 (+ 118us) client_negotiation.cc:769] Sending connection context
0812 02:37:15.458967 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 02:37:15.459137 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.3.127.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
I20260812 02:37:15.484082 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33073
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:34211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:15.606681 7040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:15.606971 7040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:15.607039 7040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:15.611186 7040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:15.611274 7040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:15.611338 7040 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 02:37:15.611362 7040 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:37:15.611411 7040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:37:15.617429 7040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:34211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.127.190:33073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:15.618659 7040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:15.619760 7040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:15.630286 7050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:15.630398 7048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:15.630384 7047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:15.630559 7040 server_base.cc:1034] running on GCE node
I20260812 02:37:15.631093 7040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:15.631933 7040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:15.633103 7040 hybrid_clock.cc:648] HybridClock initialized: now 1786502235633077 us; error 41 us; skew 500 ppm
Aug 12 02:37:15 dist-test-slave-rmhg krb5kdc[6354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502235, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:15.636030 7040 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:37:15.636976 7040 webserver.cc:460] Webserver started at http://127.3.127.131:38467/ using document root <none> and password file <none>
I20260812 02:37:15.637642 7040 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:15.637722 7040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:15.637938 7040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:15.640070 7040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "ae72c9b1260841c6af443de9be6cbbc0"
format_stamp: "Formatted at 2026-08-12 02:37:15 on dist-test-slave-rmhg"
server_key: "f2f60253a3a40690e891acdc5a888376"
server_key_iv: "f8f81c78316619840801db73508bd65a"
server_key_version: "encryptionkey@0"
I20260812 02:37:15.640653 7040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "ae72c9b1260841c6af443de9be6cbbc0"
format_stamp: "Formatted at 2026-08-12 02:37:15 on dist-test-slave-rmhg"
server_key: "f2f60253a3a40690e891acdc5a888376"
server_key_iv: "f8f81c78316619840801db73508bd65a"
server_key_version: "encryptionkey@0"
I20260812 02:37:15.644644 7040 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:37:15.647425 7062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:15.648788 7040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:37:15.648913 7040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "ae72c9b1260841c6af443de9be6cbbc0"
format_stamp: "Formatted at 2026-08-12 02:37:15 on dist-test-slave-rmhg"
server_key: "f2f60253a3a40690e891acdc5a888376"
server_key_iv: "f8f81c78316619840801db73508bd65a"
server_key_version: "encryptionkey@0"
I20260812 02:37:15.649022 7040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:15.665205 7040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:15.668547 7040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:15.668738 7040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:15.670455 7040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:15.670512 7040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:15.670579 7040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:15.670604 7040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:15.679714 7040 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:36923
I20260812 02:37:15.680073 7249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:36923 every 8 connection(s)
Aug 12 02:37:15 dist-test-slave-rmhg krb5kdc[6354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502235, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:15.680756 7040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:37:15.682289 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 7040
I20260812 02:37:15.682420 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:37:15.682760 3582 external_mini_cluster.cc:1428] Setting key d8dc2879898e2cbac2bb86f670a2a95c
I20260812 02:37:15.685281 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.671284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:46403 (local address 127.3.127.190:33073)
0812 02:37:15.671437 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:15.671442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:15.672464 (+ 1022us) server_negotiation.cc:407] Connection header received
0812 02:37:15.673706 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:15.673710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:15.673776 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:15.673880 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:15.675611 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.676443 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.677998 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.678387 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.678518 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.681368 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:15.681372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:15.681374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:15.681405 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:15.682941 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:15.683539 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:15.683542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:15.683544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:15.683678 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:15.684041 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:15.684044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:15.684045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:15.684270 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:15.684363 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:15.684942 (+ 579us) server_negotiation.cc:299] Negotiation successful
0812 02:37:15.685082 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 02:37:15.686157 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.671731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:33073 (local address 127.3.127.131:46403)
0812 02:37:15.672285 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:15.672336 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:15.673423 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:15.674069 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:15.674077 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:15.674592 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:15.675382 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.675408 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.676612 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.676616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.677842 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.677856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.678565 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.678570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.678790 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.679424 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:15.679447 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:15.681120 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:15.683066 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:15.683071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:15.683086 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:15.683448 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:15.683771 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:15.683774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:15.683776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:15.683931 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:15.684393 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:15.684399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:15.684674 (+ 275us) client_negotiation.cc:769] Sending connection context
0812 02:37:15.684867 (+ 193us) client_negotiation.cc:240] Negotiation successful
0812 02:37:15.685218 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":168,"threads_started":1}
I20260812 02:37:15.687343 7254 heartbeater.cc:347] Connected to a master server at 127.3.127.190:33073
I20260812 02:37:15.687498 7254 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:15.688097 7254 heartbeater.cc:511] Master 127.3.127.190:33073 requested a full tablet report, sending...
I20260812 02:37:15.689270 6426 ts_manager.cc:194] Registered new tserver with Master: ae72c9b1260841c6af443de9be6cbbc0 (127.3.127.131:36923)
I20260812 02:37:15.689926 6426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:46403
I20260812 02:37:15.696520 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.689785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:45767 (local address 127.3.127.190:33073)
0812 02:37:15.689907 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:15.689912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:15.690147 (+ 235us) server_negotiation.cc:407] Connection header received
0812 02:37:15.690287 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:15.690289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:15.690328 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:15.690384 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:15.690841 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.691590 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.692410 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.692633 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.692712 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.693706 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:15.693709 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:15.693711 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:15.693733 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:15.694825 (+ 1092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:15.695214 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:15.695218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:15.695220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:15.695319 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:15.695639 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:15.695641 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:15.695642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:15.695807 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:15.695900 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:15.696310 (+ 410us) server_negotiation.cc:299] Negotiation successful
0812 02:37:15.696407 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 02:37:15.696569 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.689717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:33073 (local address 127.3.127.131:45767)
0812 02:37:15.690052 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:15.690076 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:15.690174 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:15.690415 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:15.690418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:15.690612 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:15.690732 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.690738 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.691727 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.691730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.692310 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.692319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.692717 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.692719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.692788 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.693268 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:15.693279 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:15.693572 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:15.694917 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:15.694919 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:15.694921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:15.695118 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:15.695439 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:15.695441 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:15.695443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:15.695540 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:15.696023 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:15.696026 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:15.696136 (+ 110us) client_negotiation.cc:769] Sending connection context
0812 02:37:15.696245 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 02:37:15.696394 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":93,"threads_started":1}
I20260812 02:37:15.696852 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 02:37:15 dist-test-slave-rmhg krb5kdc[6354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 02:37:15 dist-test-slave-rmhg krb5kdc[6354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:15.724982 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.716413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46824 (local address 127.3.127.190:33073)
0812 02:37:15.716568 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:15.716572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:15.716585 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:37:15.716636 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:15.716638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:15.716681 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:15.716755 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:15.717608 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.718175 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.719036 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.719266 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.719356 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.722000 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:15.722004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:15.722006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:15.722037 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:15.723399 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:15.723837 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:15.723841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:15.723842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:15.723933 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:15.724183 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:15.724186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:15.724188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:15.724397 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:15.724526 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:15.724705 (+ 179us) server_negotiation.cc:299] Negotiation successful
0812 02:37:15.724819 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 02:37:15.728082 6426 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:37:15.731467 6426 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:37:15.746558 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.740495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:36923 (local address 127.0.0.1:39206)
0812 02:37:15.741015 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:15.741028 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:15.741116 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:15.742097 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:15.742100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:15.742115 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:15.742306 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.742312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.743646 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.743650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.744879 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.744888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.745978 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.745982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.746106 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.746152 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 02:37:15.746266 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 02:37:15.746393 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":59,"thread_start_us":103,"threads_started":1}
I20260812 02:37:15.746729 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.740255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:33869 (local address 127.0.0.1:53352)
0812 02:37:15.740628 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:15.740686 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:15.740845 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:15.741982 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:15.741989 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:15.742017 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:15.742256 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.742266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.743941 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.743944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.745198 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.745210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.746252 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.746256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.746344 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.746366 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 02:37:15.746442 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 02:37:15.746526 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":63,"thread_start_us":98,"threads_started":1}
I20260812 02:37:15.747051 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.741024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:41201 (local address 127.0.0.1:32878)
0812 02:37:15.741913 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:15.741926 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:15.742001 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:15.742438 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:15.742441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:15.742461 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:15.742563 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.742570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.744129 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.744132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.745500 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.745508 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.746666 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.746670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.746766 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.746787 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 02:37:15.746855 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 02:37:15.746920 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":2,"thread_start_us":152,"threads_started":1}
I20260812 02:37:15.747104 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.740424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53352 (local address 127.3.127.129:33869)
0812 02:37:15.741400 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:15.741407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:15.741430 (+ 23us) server_negotiation.cc:407] Connection header received
0812 02:37:15.741512 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:15.741518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:15.741791 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:15.742204 (+ 413us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:15.742351 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.743800 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.745340 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.746149 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.746248 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.746391 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:15.746476 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 02:37:15.746575 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":97,"threads_started":1}
I20260812 02:37:15.747246 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.740968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39206 (local address 127.3.127.131:36923)
0812 02:37:15.741427 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:15.741440 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:15.741462 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:37:15.741540 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:15.741545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:15.741785 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:15.741954 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:15.742438 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.743470 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.744999 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.745837 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.745953 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.746116 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:15.746999 (+ 883us) server_negotiation.cc:299] Negotiation successful
0812 02:37:15.747106 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":119,"threads_started":1}
I20260812 02:37:15.747511 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.741550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32878 (local address 127.3.127.130:41201)
0812 02:37:15.742007 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:15.742019 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:15.742043 (+ 24us) server_negotiation.cc:407] Connection header received
0812 02:37:15.742125 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:15.742130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:15.742299 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:15.742453 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:15.742683 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.744007 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.745614 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.746356 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.746449 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.746599 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:15.747288 (+ 689us) server_negotiation.cc:299] Negotiation successful
0812 02:37:15.747392 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":97,"threads_started":1}
I20260812 02:37:15.750404 6676 tablet_service.cc:1467] Processing CreateTablet for tablet 149d2e9c8db2408985c03d84a399fe3f (DEFAULT_TABLE table=test-table [id=246df4199c7b49b8a67a52d2fa57e374]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:15.750404 6914 tablet_service.cc:1467] Processing CreateTablet for tablet 149d2e9c8db2408985c03d84a399fe3f (DEFAULT_TABLE table=test-table [id=246df4199c7b49b8a67a52d2fa57e374]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:15.751637 6676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 149d2e9c8db2408985c03d84a399fe3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:15.751637 6914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 149d2e9c8db2408985c03d84a399fe3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:15.754954 7144 tablet_service.cc:1467] Processing CreateTablet for tablet 149d2e9c8db2408985c03d84a399fe3f (DEFAULT_TABLE table=test-table [id=246df4199c7b49b8a67a52d2fa57e374]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:15.756249 7144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 149d2e9c8db2408985c03d84a399fe3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:15.760404 7290 tablet_bootstrap.cc:492] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa: Bootstrap starting.
I20260812 02:37:15.762012 7292 tablet_bootstrap.cc:492] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5: Bootstrap starting.
I20260812 02:37:15.763607 7290 tablet_bootstrap.cc:654] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:15.764521 7290 log.cc:824] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:15.764526 7294 tablet_bootstrap.cc:492] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0: Bootstrap starting.
I20260812 02:37:15.764721 7292 tablet_bootstrap.cc:654] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:15.765584 7292 log.cc:824] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:15.766988 7294 tablet_bootstrap.cc:654] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:15.767099 7290 tablet_bootstrap.cc:492] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa: No bootstrap required, opened a new log
I20260812 02:37:15.767333 7290 ts_tablet_manager.cc:1397] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 02:37:15.767657 7292 tablet_bootstrap.cc:492] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5: No bootstrap required, opened a new log
I20260812 02:37:15.767761 7294 log.cc:824] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:15.767851 7292 ts_tablet_manager.cc:1397] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 02:37:15.769549 7294 tablet_bootstrap.cc:492] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0: No bootstrap required, opened a new log
I20260812 02:37:15.769744 7294 ts_tablet_manager.cc:1397] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 02:37:15.771564 7290 raft_consensus.cc:348] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:15.771867 7290 raft_consensus.cc:374] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:15.771937 7290 raft_consensus.cc:729] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f35f8c70bc8444ebda49652410026fa, State: Initialized, Role: FOLLOWER
I20260812 02:37:15.772140 7292 raft_consensus.cc:348] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:15.772394 7292 raft_consensus.cc:374] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:15.772449 7292 raft_consensus.cc:729] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 477261cc28a74a3cbfadc6e51e6a0bf5, State: Initialized, Role: FOLLOWER
I20260812 02:37:15.772420 7290 consensus_queue.cc:260] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:15.773224 7029 heartbeater.cc:503] Master 127.3.127.190:33073 was elected leader, sending a full tablet report...
I20260812 02:37:15.773093 7294 raft_consensus.cc:348] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:15.773378 7294 raft_consensus.cc:374] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:15.773437 7294 raft_consensus.cc:729] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae72c9b1260841c6af443de9be6cbbc0, State: Initialized, Role: FOLLOWER
I20260812 02:37:15.773597 7290 ts_tablet_manager.cc:1428] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260812 02:37:15.773842 7294 consensus_queue.cc:260] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:15.772912 7292 consensus_queue.cc:260] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:15.774531 7254 heartbeater.cc:503] Master 127.3.127.190:33073 was elected leader, sending a full tablet report...
I20260812 02:37:15.774827 7294 ts_tablet_manager.cc:1428] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 02:37:15.775207 7292 ts_tablet_manager.cc:1428] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 02:37:15.775588 6793 heartbeater.cc:503] Master 127.3.127.190:33073 was elected leader, sending a full tablet report...
I20260812 02:37:15.889931 7298 raft_consensus.cc:482] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:37:15.890206 7298 raft_consensus.cc:504] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:15.891573 7298 leader_election.cc:288] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 477261cc28a74a3cbfadc6e51e6a0bf5 (127.3.127.129:33869), ae72c9b1260841c6af443de9be6cbbc0 (127.3.127.131:36923)
I20260812 02:37:15.895821 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.892393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:59569 (local address 127.3.127.129:33869)
0812 02:37:15.892540 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:15.892544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:15.892570 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:37:15.892628 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:15.892631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:15.892683 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:15.892757 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:15.893082 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.893751 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.894835 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.895278 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.895394 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.895453 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:15.895629 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 02:37:15.895720 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 02:37:15.895821 6850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.891853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:33869 (local address 127.3.127.130:59569)
0812 02:37:15.892168 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:15.892184 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:15.892320 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:15.892818 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:15.892821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:15.892834 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:15.892966 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.892973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.893866 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.893869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.894697 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.894707 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.895401 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.895405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.895512 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.895531 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 02:37:15.895597 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 02:37:15.895678 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":2}
I20260812 02:37:15.896257 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.892061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:59541 (local address 127.3.127.131:36923)
0812 02:37:15.892174 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:15.892180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:15.892695 (+ 515us) server_negotiation.cc:407] Connection header received
0812 02:37:15.892792 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:15.892795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:15.892847 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:15.892912 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:15.893680 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.894193 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.895321 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.895722 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.895830 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.895883 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:15.896076 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 02:37:15.896150 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 02:37:15.896279 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.891853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:36923 (local address 127.3.127.130:59541)
0812 02:37:15.892587 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:15.892598 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:15.892702 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:15.892955 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:15.892958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:15.892979 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:15.893547 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.893558 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.894308 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.894311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.895189 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.895199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.895853 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.895857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.895958 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.895984 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 02:37:15.896058 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 02:37:15.896126 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":427,"thread_start_us":166,"threads_started":1}
I20260812 02:37:15.896572 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "149d2e9c8db2408985c03d84a399fe3f" candidate_uuid: "8f35f8c70bc8444ebda49652410026fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" is_pre_election: true
I20260812 02:37:15.896927 6708 raft_consensus.cc:2454] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f35f8c70bc8444ebda49652410026fa in term 0.
I20260812 02:37:15.896987 7176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "149d2e9c8db2408985c03d84a399fe3f" candidate_uuid: "8f35f8c70bc8444ebda49652410026fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae72c9b1260841c6af443de9be6cbbc0" is_pre_election: true
I20260812 02:37:15.897328 7176 raft_consensus.cc:2454] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f35f8c70bc8444ebda49652410026fa in term 0.
I20260812 02:37:15.897477 6837 leader_election.cc:302] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 477261cc28a74a3cbfadc6e51e6a0bf5, 8f35f8c70bc8444ebda49652410026fa; no voters:
I20260812 02:37:15.897847 7298 raft_consensus.cc:2781] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:37:15.897939 7298 raft_consensus.cc:482] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:37:15.897989 7298 raft_consensus.cc:3037] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:15.899052 7298 raft_consensus.cc:504] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:15.899398 7298 leader_election.cc:288] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa [CANDIDATE]: Term 1 election: Requested vote from peers 477261cc28a74a3cbfadc6e51e6a0bf5 (127.3.127.129:33869), ae72c9b1260841c6af443de9be6cbbc0 (127.3.127.131:36923)
I20260812 02:37:15.899809 7176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "149d2e9c8db2408985c03d84a399fe3f" candidate_uuid: "8f35f8c70bc8444ebda49652410026fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae72c9b1260841c6af443de9be6cbbc0"
I20260812 02:37:15.899809 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "149d2e9c8db2408985c03d84a399fe3f" candidate_uuid: "8f35f8c70bc8444ebda49652410026fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5"
I20260812 02:37:15.899945 7176 raft_consensus.cc:3037] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:15.899945 6708 raft_consensus.cc:3037] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:15.901326 7176 raft_consensus.cc:2454] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f35f8c70bc8444ebda49652410026fa in term 1.
I20260812 02:37:15.901326 6708 raft_consensus.cc:2454] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f35f8c70bc8444ebda49652410026fa in term 1.
I20260812 02:37:15.901669 6837 leader_election.cc:302] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 477261cc28a74a3cbfadc6e51e6a0bf5, 8f35f8c70bc8444ebda49652410026fa; no voters:
I20260812 02:37:15.901870 7298 raft_consensus.cc:2781] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:15.902119 7298 raft_consensus.cc:686] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa [term 1 LEADER]: Becoming Leader. State: Replica: 8f35f8c70bc8444ebda49652410026fa, State: Running, Role: LEADER
I20260812 02:37:15.902496 7298 consensus_queue.cc:237] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:15.906322 6427 catalog_manager.cc:5495] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f35f8c70bc8444ebda49652410026fa (127.3.127.130). New cstate: current_term: 1 leader_uuid: "8f35f8c70bc8444ebda49652410026fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } health_report { overall_health: UNKNOWN } } }
W20260812 02:37:15.932087 7255 tablet.cc:2369] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:37:15.946313 7030 tablet.cc:2369] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 02:37:15 dist-test-slave-rmhg krb5kdc[6354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502234, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:15.960088 7317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.951104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:33073 (local address 127.0.0.1:46828)
0812 02:37:15.951419 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:15.951432 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:15.951556 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:15.951834 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:15.951837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:15.952368 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:15.952488 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.952496 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.953347 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.953349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.953978 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:15.953987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.954453 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.954456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:15.954537 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.955130 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:15.955143 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:15.956689 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:15.958298 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:15.958313 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:15.958322 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:15.958661 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:15.958949 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:15.958952 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:15.958954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:15.959047 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:15.959511 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:15.959518 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:15.959640 (+ 122us) client_negotiation.cc:769] Sending connection context
0812 02:37:15.959758 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 02:37:15.959922 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":118,"threads_started":1}
I20260812 02:37:15.960124 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:15.951187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46828 (local address 127.3.127.190:33073)
0812 02:37:15.951337 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:15.951342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:15.951510 (+ 168us) server_negotiation.cc:407] Connection header received
0812 02:37:15.951639 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:15.951641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:15.951684 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:15.951763 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:15.952631 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.953220 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.954112 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:15.954344 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:15.954435 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:15.956861 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:15.956864 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:15.956866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:15.956889 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:15.958182 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:15.958759 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:15.958762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:15.958763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:15.958844 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:15.959165 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:15.959168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:15.959169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:15.959383 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:15.959473 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:15.959840 (+ 367us) server_negotiation.cc:299] Negotiation successful
0812 02:37:15.960001 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":64}
W20260812 02:37:15.962285 6796 tablet.cc:2369] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:37:15.964696 6427 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:46828:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 02:37:15.965544 6427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:37:15.970870 6914 tablet_service.cc:1467] Processing CreateTablet for tablet 616208c9db89492e8bf17fa893ef3278 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3c5c36323ec43b797349b06558c193f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:37:15.970854 7144 tablet_service.cc:1467] Processing CreateTablet for tablet 616208c9db89492e8bf17fa893ef3278 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3c5c36323ec43b797349b06558c193f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:37:15.970855 6676 tablet_service.cc:1467] Processing CreateTablet for tablet 616208c9db89492e8bf17fa893ef3278 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3c5c36323ec43b797349b06558c193f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:37:15.971606 6914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 616208c9db89492e8bf17fa893ef3278. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:15.971609 6676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 616208c9db89492e8bf17fa893ef3278. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:15.971642 7144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 616208c9db89492e8bf17fa893ef3278. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:15.974243 7290 tablet_bootstrap.cc:492] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa: Bootstrap starting.
I20260812 02:37:15.974529 7292 tablet_bootstrap.cc:492] T 616208c9db89492e8bf17fa893ef3278 P 477261cc28a74a3cbfadc6e51e6a0bf5: Bootstrap starting.
I20260812 02:37:15.974530 7294 tablet_bootstrap.cc:492] T 616208c9db89492e8bf17fa893ef3278 P ae72c9b1260841c6af443de9be6cbbc0: Bootstrap starting.
I20260812 02:37:15.975304 7290 tablet_bootstrap.cc:654] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:15.975508 7292 tablet_bootstrap.cc:654] T 616208c9db89492e8bf17fa893ef3278 P 477261cc28a74a3cbfadc6e51e6a0bf5: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:15.975508 7294 tablet_bootstrap.cc:654] T 616208c9db89492e8bf17fa893ef3278 P ae72c9b1260841c6af443de9be6cbbc0: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:15.976836 7292 tablet_bootstrap.cc:492] T 616208c9db89492e8bf17fa893ef3278 P 477261cc28a74a3cbfadc6e51e6a0bf5: No bootstrap required, opened a new log
I20260812 02:37:15.976841 7290 tablet_bootstrap.cc:492] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa: No bootstrap required, opened a new log
I20260812 02:37:15.976929 7290 ts_tablet_manager.cc:1397] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:37:15.976929 7292 ts_tablet_manager.cc:1397] T 616208c9db89492e8bf17fa893ef3278 P 477261cc28a74a3cbfadc6e51e6a0bf5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 02:37:15.977126 7294 tablet_bootstrap.cc:492] T 616208c9db89492e8bf17fa893ef3278 P ae72c9b1260841c6af443de9be6cbbc0: No bootstrap required, opened a new log
I20260812 02:37:15.977236 7294 ts_tablet_manager.cc:1397] T 616208c9db89492e8bf17fa893ef3278 P ae72c9b1260841c6af443de9be6cbbc0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:37:15.977475 7290 raft_consensus.cc:348] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:15.977476 7292 raft_consensus.cc:348] T 616208c9db89492e8bf17fa893ef3278 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:15.977605 7290 raft_consensus.cc:374] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:15.977605 7292 raft_consensus.cc:374] T 616208c9db89492e8bf17fa893ef3278 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:15.977648 7292 raft_consensus.cc:729] T 616208c9db89492e8bf17fa893ef3278 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 477261cc28a74a3cbfadc6e51e6a0bf5, State: Initialized, Role: FOLLOWER
I20260812 02:37:15.977648 7290 raft_consensus.cc:729] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f35f8c70bc8444ebda49652410026fa, State: Initialized, Role: FOLLOWER
I20260812 02:37:15.977684 7294 raft_consensus.cc:348] T 616208c9db89492e8bf17fa893ef3278 P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:15.977790 7294 raft_consensus.cc:374] T 616208c9db89492e8bf17fa893ef3278 P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:15.977771 7292 consensus_queue.cc:260] T 616208c9db89492e8bf17fa893ef3278 P 477261cc28a74a3cbfadc6e51e6a0bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:15.977773 7290 consensus_queue.cc:260] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:15.977837 7294 raft_consensus.cc:729] T 616208c9db89492e8bf17fa893ef3278 P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae72c9b1260841c6af443de9be6cbbc0, State: Initialized, Role: FOLLOWER
I20260812 02:37:15.977952 7294 consensus_queue.cc:260] T 616208c9db89492e8bf17fa893ef3278 P ae72c9b1260841c6af443de9be6cbbc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:15.978058 7290 ts_tablet_manager.cc:1428] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:37:15.978196 7294 ts_tablet_manager.cc:1428] T 616208c9db89492e8bf17fa893ef3278 P ae72c9b1260841c6af443de9be6cbbc0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:37:15.978119 7300 tablet_replica.cc:439] T 616208c9db89492e8bf17fa893ef3278 P 477261cc28a74a3cbfadc6e51e6a0bf5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } } }
I20260812 02:37:15.978276 7300 tablet_replica.cc:442] T 616208c9db89492e8bf17fa893ef3278 P 477261cc28a74a3cbfadc6e51e6a0bf5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:37:15.978255 7298 tablet_replica.cc:439] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } } }
I20260812 02:37:15.978390 7298 tablet_replica.cc:442] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:37:15.978466 7299 tablet_replica.cc:439] T 616208c9db89492e8bf17fa893ef3278 P ae72c9b1260841c6af443de9be6cbbc0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } } }
I20260812 02:37:15.978605 7299 tablet_replica.cc:442] T 616208c9db89492e8bf17fa893ef3278 P ae72c9b1260841c6af443de9be6cbbc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:37:15.978652 7292 ts_tablet_manager.cc:1428] T 616208c9db89492e8bf17fa893ef3278 P 477261cc28a74a3cbfadc6e51e6a0bf5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 02:37:16.076490 7298 raft_consensus.cc:482] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:37:16.076643 7298 raft_consensus.cc:504] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:16.077140 7298 leader_election.cc:288] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 477261cc28a74a3cbfadc6e51e6a0bf5 (127.3.127.129:33869), ae72c9b1260841c6af443de9be6cbbc0 (127.3.127.131:36923)
I20260812 02:37:16.077766 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616208c9db89492e8bf17fa893ef3278" candidate_uuid: "8f35f8c70bc8444ebda49652410026fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" is_pre_election: true
I20260812 02:37:16.077765 7176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616208c9db89492e8bf17fa893ef3278" candidate_uuid: "8f35f8c70bc8444ebda49652410026fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae72c9b1260841c6af443de9be6cbbc0" is_pre_election: true
I20260812 02:37:16.077948 7176 raft_consensus.cc:2454] T 616208c9db89492e8bf17fa893ef3278 P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f35f8c70bc8444ebda49652410026fa in term 0.
I20260812 02:37:16.077945 6708 raft_consensus.cc:2454] T 616208c9db89492e8bf17fa893ef3278 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f35f8c70bc8444ebda49652410026fa in term 0.
I20260812 02:37:16.078332 6838 leader_election.cc:302] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 477261cc28a74a3cbfadc6e51e6a0bf5, 8f35f8c70bc8444ebda49652410026fa, ae72c9b1260841c6af443de9be6cbbc0; no voters:
I20260812 02:37:16.078562 7298 raft_consensus.cc:2781] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:37:16.078630 7298 raft_consensus.cc:482] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:37:16.078656 7298 raft_consensus.cc:3037] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:16.079741 7298 raft_consensus.cc:504] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:16.080116 7298 leader_election.cc:288] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa [CANDIDATE]: Term 1 election: Requested vote from peers 477261cc28a74a3cbfadc6e51e6a0bf5 (127.3.127.129:33869), ae72c9b1260841c6af443de9be6cbbc0 (127.3.127.131:36923)
I20260812 02:37:16.080575 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616208c9db89492e8bf17fa893ef3278" candidate_uuid: "8f35f8c70bc8444ebda49652410026fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5"
I20260812 02:37:16.080616 7176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616208c9db89492e8bf17fa893ef3278" candidate_uuid: "8f35f8c70bc8444ebda49652410026fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae72c9b1260841c6af443de9be6cbbc0"
I20260812 02:37:16.080734 7176 raft_consensus.cc:3037] T 616208c9db89492e8bf17fa893ef3278 P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:16.080734 6708 raft_consensus.cc:3037] T 616208c9db89492e8bf17fa893ef3278 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:16.081959 6708 raft_consensus.cc:2454] T 616208c9db89492e8bf17fa893ef3278 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f35f8c70bc8444ebda49652410026fa in term 1.
I20260812 02:37:16.081959 7176 raft_consensus.cc:2454] T 616208c9db89492e8bf17fa893ef3278 P ae72c9b1260841c6af443de9be6cbbc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f35f8c70bc8444ebda49652410026fa in term 1.
I20260812 02:37:16.082324 6837 leader_election.cc:302] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 477261cc28a74a3cbfadc6e51e6a0bf5, 8f35f8c70bc8444ebda49652410026fa; no voters:
I20260812 02:37:16.082530 7298 raft_consensus.cc:2781] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:16.082638 7298 raft_consensus.cc:686] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa [term 1 LEADER]: Becoming Leader. State: Replica: 8f35f8c70bc8444ebda49652410026fa, State: Running, Role: LEADER
I20260812 02:37:16.082782 7298 consensus_queue.cc:237] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:16.083489 7302 tablet_replica.cc:439] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa: TxnStatusTablet state changed. Reason: New leader 8f35f8c70bc8444ebda49652410026fa. Latest consensus state: current_term: 1 leader_uuid: "8f35f8c70bc8444ebda49652410026fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } } }
I20260812 02:37:16.083612 7302 tablet_replica.cc:442] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:37:16.083940 7325 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 02:37:16.084201 7325 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 02:37:16.084695 6434 catalog_manager.cc:5495] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f35f8c70bc8444ebda49652410026fa (127.3.127.130). New cstate: current_term: 1 leader_uuid: "8f35f8c70bc8444ebda49652410026fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } health_report { overall_health: UNKNOWN } } }
I20260812 02:37:16.108471 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:16.104850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32894 (local address 127.3.127.130:41201)
0812 02:37:16.104997 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:16.105001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:16.105017 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:37:16.105208 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:16.105210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:16.105293 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:16.105388 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:16.105646 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.106175 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.107024 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.107246 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.107337 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:16.107628 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:16.108069 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:16.108183 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:16.108262 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 02:37:16.108323 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 02:37:16.108471 7317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:16.104693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:41201 (local address 127.0.0.1:32894)
0812 02:37:16.104897 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:16.104916 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:16.105030 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:16.105426 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:16.105429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:16.105437 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:37:16.105544 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:16.105550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.106280 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.106283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:16.106912 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:16.106921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.107347 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.107350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:16.107440 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:16.107486 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:37:16.108172 (+ 686us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:37:16.108189 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 02:37:16.108251 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 02:37:16.108338 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 02:37:16.114426 6708 raft_consensus.cc:1261] T 616208c9db89492e8bf17fa893ef3278 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 1 FOLLOWER]: Refusing update from remote peer 8f35f8c70bc8444ebda49652410026fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:16.114411 7176 raft_consensus.cc:1261] T 616208c9db89492e8bf17fa893ef3278 P ae72c9b1260841c6af443de9be6cbbc0 [term 1 FOLLOWER]: Refusing update from remote peer 8f35f8c70bc8444ebda49652410026fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:16.115195 7298 consensus_queue.cc:1035] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:16.115697 7298 consensus_queue.cc:1035] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:16.120290 7299 tablet_replica.cc:439] T 616208c9db89492e8bf17fa893ef3278 P ae72c9b1260841c6af443de9be6cbbc0: TxnStatusTablet state changed. Reason: New leader 8f35f8c70bc8444ebda49652410026fa. Latest consensus state: current_term: 1 leader_uuid: "8f35f8c70bc8444ebda49652410026fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } } }
I20260812 02:37:16.120442 7299 tablet_replica.cc:442] T 616208c9db89492e8bf17fa893ef3278 P ae72c9b1260841c6af443de9be6cbbc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:37:16.120594 7300 tablet_replica.cc:439] T 616208c9db89492e8bf17fa893ef3278 P 477261cc28a74a3cbfadc6e51e6a0bf5: TxnStatusTablet state changed. Reason: New leader 8f35f8c70bc8444ebda49652410026fa. Latest consensus state: current_term: 1 leader_uuid: "8f35f8c70bc8444ebda49652410026fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } } }
I20260812 02:37:16.120746 7300 tablet_replica.cc:442] T 616208c9db89492e8bf17fa893ef3278 P 477261cc28a74a3cbfadc6e51e6a0bf5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:37:16.124588 7299 tablet_replica.cc:439] T 616208c9db89492e8bf17fa893ef3278 P ae72c9b1260841c6af443de9be6cbbc0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f35f8c70bc8444ebda49652410026fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } } }
I20260812 02:37:16.124715 7299 tablet_replica.cc:442] T 616208c9db89492e8bf17fa893ef3278 P ae72c9b1260841c6af443de9be6cbbc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:37:16.125442 7300 tablet_replica.cc:439] T 616208c9db89492e8bf17fa893ef3278 P 477261cc28a74a3cbfadc6e51e6a0bf5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f35f8c70bc8444ebda49652410026fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } } }
I20260812 02:37:16.125588 7300 tablet_replica.cc:442] T 616208c9db89492e8bf17fa893ef3278 P 477261cc28a74a3cbfadc6e51e6a0bf5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:37:16.125557 7339 tablet_replica.cc:439] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f35f8c70bc8444ebda49652410026fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } } }
I20260812 02:37:16.125859 7339 tablet_replica.cc:442] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:37:16.126157 7302 tablet_replica.cc:439] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f35f8c70bc8444ebda49652410026fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } } }
I20260812 02:37:16.126273 7302 tablet_replica.cc:442] T 616208c9db89492e8bf17fa893ef3278 P 8f35f8c70bc8444ebda49652410026fa: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:37:16.127609 7334 mvcc.cc:205] Tried to move back new op lower bound from 7317513159116615680 to 7317513158996475904. Current Snapshot: MvccSnapshot[applied={T|T < 7317513159116615680}]
I20260812 02:37:16.127978 7333 mvcc.cc:205] Tried to move back new op lower bound from 7317513159116615680 to 7317513158996475904. Current Snapshot: MvccSnapshot[applied={T|T < 7317513159116615680}]
I20260812 02:37:16.138228 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:16.134807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32906 (local address 127.3.127.130:41201)
0812 02:37:16.134934 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:16.134938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:16.134956 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:37:16.135068 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:16.135071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:16.135123 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:16.135208 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:16.135410 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.136213 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.137039 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.137419 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.137523 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:16.137792 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:16.137883 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:16.137950 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:16.138020 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 02:37:16.138082 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 02:37:16.151062 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:16.147500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:57487 (local address 127.3.127.130:41201)
0812 02:37:16.147667 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:16.147671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:16.147885 (+ 214us) server_negotiation.cc:407] Connection header received
0812 02:37:16.148011 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:16.148014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:16.148055 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:16.148116 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:16.148371 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.148885 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.149908 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.150160 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.150244 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:16.150578 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:16.150665 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:16.150723 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:16.150807 (+ 84us) server_negotiation.cc:299] Negotiation successful
0812 02:37:16.150888 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":44}
I20260812 02:37:16.151070 7348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:16.147407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:41201 (local address 127.3.127.130:57487)
0812 02:37:16.147805 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:16.147817 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:16.147925 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:16.148135 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:16.148138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:16.148145 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:37:16.148281 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:16.148287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.149002 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.149006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:16.149801 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:16.149813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.150266 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.150269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:16.150364 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:16.150407 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:37:16.150736 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:37:16.150742 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 02:37:16.150799 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 02:37:16.150916 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":45,"thread_start_us":107,"threads_started":1}
I20260812 02:37:16.159838 7176 raft_consensus.cc:1261] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0 [term 1 FOLLOWER]: Refusing update from remote peer 8f35f8c70bc8444ebda49652410026fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:16.159936 6708 raft_consensus.cc:1261] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 1 FOLLOWER]: Refusing update from remote peer 8f35f8c70bc8444ebda49652410026fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:16.160318 7298 consensus_queue.cc:1035] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:16.160567 7339 consensus_queue.cc:1035] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:16.167112 7330 mvcc.cc:205] Tried to move back new op lower bound from 7317513159306014720 to 7317513158259838976. Current Snapshot: MvccSnapshot[applied={T|T < 7317513159306014720}]
I20260812 02:37:16.211112 6426 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:16.211333 6426 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:16.214495 6426 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c: Sending DeleteTablet for 3 replicas of tablet 149d2e9c8db2408985c03d84a399fe3f
I20260812 02:37:16.215662 6676 tablet_service.cc:1514] Processing DeleteTablet for tablet 149d2e9c8db2408985c03d84a399fe3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:16 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:53352
I20260812 02:37:16.216281 7354 tablet_replica.cc:330] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5: stopping tablet replica
I20260812 02:37:16.216424 7144 tablet_service.cc:1514] Processing DeleteTablet for tablet 149d2e9c8db2408985c03d84a399fe3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:16 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:39206
I20260812 02:37:16.216882 7354 raft_consensus.cc:2229] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:16.217381 7354 raft_consensus.cc:2258] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:16.218097 6914 tablet_service.cc:1514] Processing DeleteTablet for tablet 149d2e9c8db2408985c03d84a399fe3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:16 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:32878
I20260812 02:37:16.218163 6426 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:37:16.218705 6426 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:37:16.219643 7355 tablet_replica.cc:330] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0: stopping tablet replica
I20260812 02:37:16.219985 7355 raft_consensus.cc:2229] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:16.220201 7355 raft_consensus.cc:2258] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:16.221490 7354 ts_tablet_manager.cc:1905] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:16.224707 7354 ts_tablet_manager.cc:1918] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:37:16.224813 7354 log.cc:1196] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-0/wal/wals/149d2e9c8db2408985c03d84a399fe3f
I20260812 02:37:16.225090 7354 ts_tablet_manager.cc:1939] T 149d2e9c8db2408985c03d84a399fe3f P 477261cc28a74a3cbfadc6e51e6a0bf5: Deleting consensus metadata
I20260812 02:37:16.225379 7356 tablet_replica.cc:330] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa: stopping tablet replica
I20260812 02:37:16.225762 7356 raft_consensus.cc:2229] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:37:16.225769 6676 tablet_service.cc:1467] Processing CreateTablet for tablet 4569eb380a76498dab96b172aa142d72 (DEFAULT_TABLE table=test-table [id=998c17848f3f448485c46968151d3d6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:16.226076 6676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4569eb380a76498dab96b172aa142d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:16.226192 7356 raft_consensus.cc:2258] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:16.226624 7355 ts_tablet_manager.cc:1905] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:16.228942 7144 tablet_service.cc:1467] Processing CreateTablet for tablet 4569eb380a76498dab96b172aa142d72 (DEFAULT_TABLE table=test-table [id=998c17848f3f448485c46968151d3d6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:16.229292 7144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4569eb380a76498dab96b172aa142d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:16.230271 7355 ts_tablet_manager.cc:1918] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:37:16.230371 7355 log.cc:1196] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/149d2e9c8db2408985c03d84a399fe3f
I20260812 02:37:16.230739 7355 ts_tablet_manager.cc:1939] T 149d2e9c8db2408985c03d84a399fe3f P ae72c9b1260841c6af443de9be6cbbc0: Deleting consensus metadata
I20260812 02:37:16.231593 6914 tablet_service.cc:1467] Processing CreateTablet for tablet 4569eb380a76498dab96b172aa142d72 (DEFAULT_TABLE table=test-table [id=998c17848f3f448485c46968151d3d6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:16.231922 6914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4569eb380a76498dab96b172aa142d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:16.233337 7356 ts_tablet_manager.cc:1905] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:16.234521 7290 tablet_bootstrap.cc:492] T 4569eb380a76498dab96b172aa142d72 P 8f35f8c70bc8444ebda49652410026fa: Bootstrap starting.
I20260812 02:37:16.234817 7292 tablet_bootstrap.cc:492] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5: Bootstrap starting.
I20260812 02:37:16.235526 7290 tablet_bootstrap.cc:654] T 4569eb380a76498dab96b172aa142d72 P 8f35f8c70bc8444ebda49652410026fa: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:16.235812 7292 tablet_bootstrap.cc:654] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:16.236539 7356 ts_tablet_manager.cc:1918] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:37:16.236529 6407 catalog_manager.cc:4841] TS ae72c9b1260841c6af443de9be6cbbc0 (127.3.127.131:36923): tablet 149d2e9c8db2408985c03d84a399fe3f (table test-table [id=246df4199c7b49b8a67a52d2fa57e374]) successfully deleted
I20260812 02:37:16.236649 7356 log.cc:1196] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/149d2e9c8db2408985c03d84a399fe3f
I20260812 02:37:16.236868 6406 catalog_manager.cc:4841] TS 477261cc28a74a3cbfadc6e51e6a0bf5 (127.3.127.129:33869): tablet 149d2e9c8db2408985c03d84a399fe3f (table test-table [id=246df4199c7b49b8a67a52d2fa57e374]) successfully deleted
I20260812 02:37:16.236984 7356 ts_tablet_manager.cc:1939] T 149d2e9c8db2408985c03d84a399fe3f P 8f35f8c70bc8444ebda49652410026fa: Deleting consensus metadata
I20260812 02:37:16.237358 7294 tablet_bootstrap.cc:492] T 4569eb380a76498dab96b172aa142d72 P ae72c9b1260841c6af443de9be6cbbc0: Bootstrap starting.
I20260812 02:37:16.238315 6406 catalog_manager.cc:4841] TS 8f35f8c70bc8444ebda49652410026fa (127.3.127.130:41201): tablet 149d2e9c8db2408985c03d84a399fe3f (table test-table [id=246df4199c7b49b8a67a52d2fa57e374]) successfully deleted
I20260812 02:37:16.238440 7294 tablet_bootstrap.cc:654] T 4569eb380a76498dab96b172aa142d72 P ae72c9b1260841c6af443de9be6cbbc0: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:16.238968 7290 tablet_bootstrap.cc:492] T 4569eb380a76498dab96b172aa142d72 P 8f35f8c70bc8444ebda49652410026fa: No bootstrap required, opened a new log
I20260812 02:37:16.239056 7290 ts_tablet_manager.cc:1397] T 4569eb380a76498dab96b172aa142d72 P 8f35f8c70bc8444ebda49652410026fa: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260812 02:37:16.239506 7290 raft_consensus.cc:348] T 4569eb380a76498dab96b172aa142d72 P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:16.239627 7290 raft_consensus.cc:374] T 4569eb380a76498dab96b172aa142d72 P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:16.239655 7290 raft_consensus.cc:729] T 4569eb380a76498dab96b172aa142d72 P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f35f8c70bc8444ebda49652410026fa, State: Initialized, Role: FOLLOWER
I20260812 02:37:16.239784 7290 consensus_queue.cc:260] T 4569eb380a76498dab96b172aa142d72 P 8f35f8c70bc8444ebda49652410026fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:16.239861 7292 tablet_bootstrap.cc:492] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5: No bootstrap required, opened a new log
I20260812 02:37:16.239961 7292 ts_tablet_manager.cc:1397] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260812 02:37:16.240193 7294 tablet_bootstrap.cc:492] T 4569eb380a76498dab96b172aa142d72 P ae72c9b1260841c6af443de9be6cbbc0: No bootstrap required, opened a new log
I20260812 02:37:16.240276 7294 ts_tablet_manager.cc:1397] T 4569eb380a76498dab96b172aa142d72 P ae72c9b1260841c6af443de9be6cbbc0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:37:16.240468 7292 raft_consensus.cc:348] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:16.240581 7292 raft_consensus.cc:374] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:16.240612 7292 raft_consensus.cc:729] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 477261cc28a74a3cbfadc6e51e6a0bf5, State: Initialized, Role: FOLLOWER
I20260812 02:37:16.240736 7292 consensus_queue.cc:260] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:16.240747 7294 raft_consensus.cc:348] T 4569eb380a76498dab96b172aa142d72 P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:16.240847 7294 raft_consensus.cc:374] T 4569eb380a76498dab96b172aa142d72 P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:16.240876 7294 raft_consensus.cc:729] T 4569eb380a76498dab96b172aa142d72 P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae72c9b1260841c6af443de9be6cbbc0, State: Initialized, Role: FOLLOWER
I20260812 02:37:16.240994 7294 consensus_queue.cc:260] T 4569eb380a76498dab96b172aa142d72 P ae72c9b1260841c6af443de9be6cbbc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:16.241307 7294 ts_tablet_manager.cc:1428] T 4569eb380a76498dab96b172aa142d72 P ae72c9b1260841c6af443de9be6cbbc0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:37:16.241714 7292 ts_tablet_manager.cc:1428] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 02:37:16.242959 7290 ts_tablet_manager.cc:1428] T 4569eb380a76498dab96b172aa142d72 P 8f35f8c70bc8444ebda49652410026fa: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260812 02:37:16.305966 7300 raft_consensus.cc:482] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:37:16.306157 7300 raft_consensus.cc:504] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:16.307526 7300 leader_election.cc:288] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f35f8c70bc8444ebda49652410026fa (127.3.127.130:41201), ae72c9b1260841c6af443de9be6cbbc0 (127.3.127.131:36923)
I20260812 02:37:16.312814 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:16.307949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:50605 (local address 127.3.127.131:36923)
0812 02:37:16.308147 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:16.308152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:16.308442 (+ 290us) server_negotiation.cc:407] Connection header received
0812 02:37:16.308573 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:16.308576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:16.308622 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:16.308677 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:16.309055 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.309611 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.311506 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.312174 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.312339 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:16.312400 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:16.312612 (+ 212us) server_negotiation.cc:299] Negotiation successful
0812 02:37:16.312701 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260812 02:37:16.312901 7367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:16.307808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:36923 (local address 127.3.127.129:50605)
0812 02:37:16.308341 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:16.308354 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:16.308453 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:16.308811 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:16.308814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:16.308832 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:16.308941 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:16.308945 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.309737 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.309741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:16.311365 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:16.311380 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.312336 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.312341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:16.312449 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:16.312475 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 02:37:16.312552 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 02:37:16.312641 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":102,"thread_start_us":109,"threads_started":1}
I20260812 02:37:16.313726 7362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:16.307761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:41201 (local address 127.3.127.129:38337)
0812 02:37:16.308132 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:16.308146 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:16.308258 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:16.308514 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:16.308517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:16.308533 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:16.308694 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:16.308703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.309782 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.309785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:16.311371 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:16.311380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.313286 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.313290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:16.313397 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:16.313423 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 02:37:16.313500 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 02:37:16.313578 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":162,"thread_start_us":77,"threads_started":1}
I20260812 02:37:16.313736 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:16.308138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:38337 (local address 127.3.127.130:41201)
0812 02:37:16.308253 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:16.308257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:16.308272 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:37:16.308360 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:16.308364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:16.308412 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:16.308483 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:16.308810 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.309633 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.312496 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.313131 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.313276 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:16.313331 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:16.313524 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 02:37:16.313603 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 02:37:16.313730 7176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4569eb380a76498dab96b172aa142d72" candidate_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae72c9b1260841c6af443de9be6cbbc0" is_pre_election: true
I20260812 02:37:16.313901 7176 raft_consensus.cc:2454] T 4569eb380a76498dab96b172aa142d72 P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 477261cc28a74a3cbfadc6e51e6a0bf5 in term 0.
I20260812 02:37:16.314365 6946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4569eb380a76498dab96b172aa142d72" candidate_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f35f8c70bc8444ebda49652410026fa" is_pre_election: true
I20260812 02:37:16.314379 6590 leader_election.cc:302] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 477261cc28a74a3cbfadc6e51e6a0bf5, ae72c9b1260841c6af443de9be6cbbc0; no voters:
I20260812 02:37:16.314606 6946 raft_consensus.cc:2454] T 4569eb380a76498dab96b172aa142d72 P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 477261cc28a74a3cbfadc6e51e6a0bf5 in term 0.
I20260812 02:37:16.314687 7300 raft_consensus.cc:2781] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:37:16.314749 7300 raft_consensus.cc:482] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:37:16.314788 7300 raft_consensus.cc:3037] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:16.316200 7300 raft_consensus.cc:504] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:16.316578 7300 leader_election.cc:288] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 [CANDIDATE]: Term 1 election: Requested vote from peers 8f35f8c70bc8444ebda49652410026fa (127.3.127.130:41201), ae72c9b1260841c6af443de9be6cbbc0 (127.3.127.131:36923)
I20260812 02:37:16.316992 7176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4569eb380a76498dab96b172aa142d72" candidate_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae72c9b1260841c6af443de9be6cbbc0"
I20260812 02:37:16.317011 6946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4569eb380a76498dab96b172aa142d72" candidate_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f35f8c70bc8444ebda49652410026fa"
I20260812 02:37:16.317113 7176 raft_consensus.cc:3037] T 4569eb380a76498dab96b172aa142d72 P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:16.317128 6946 raft_consensus.cc:3037] T 4569eb380a76498dab96b172aa142d72 P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:16.318162 7176 raft_consensus.cc:2454] T 4569eb380a76498dab96b172aa142d72 P ae72c9b1260841c6af443de9be6cbbc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 477261cc28a74a3cbfadc6e51e6a0bf5 in term 1.
I20260812 02:37:16.318375 6946 raft_consensus.cc:2454] T 4569eb380a76498dab96b172aa142d72 P 8f35f8c70bc8444ebda49652410026fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 477261cc28a74a3cbfadc6e51e6a0bf5 in term 1.
I20260812 02:37:16.318434 6590 leader_election.cc:302] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 477261cc28a74a3cbfadc6e51e6a0bf5, ae72c9b1260841c6af443de9be6cbbc0; no voters:
I20260812 02:37:16.318601 7300 raft_consensus.cc:2781] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:16.318830 7300 raft_consensus.cc:686] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 1 LEADER]: Becoming Leader. State: Replica: 477261cc28a74a3cbfadc6e51e6a0bf5, State: Running, Role: LEADER
I20260812 02:37:16.319106 7300 consensus_queue.cc:237] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:16.321600 6426 catalog_manager.cc:5495] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 477261cc28a74a3cbfadc6e51e6a0bf5 (127.3.127.129). New cstate: current_term: 1 leader_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } health_report { overall_health: UNKNOWN } } }
I20260812 02:37:16.355875 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:16.352010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53358 (local address 127.3.127.129:33869)
0812 02:37:16.352137 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:16.352142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:16.352156 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:37:16.352290 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:16.352293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:16.352350 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:16.352455 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:16.352705 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.353349 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.354298 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.354528 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.354696 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:16.354964 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:16.355448 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:16.355557 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:16.355645 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 02:37:16.355711 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 02:37:16.368762 7377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:16.364524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:41201 (local address 127.3.127.129:43279)
0812 02:37:16.364947 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:16.364961 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:16.365083 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:16.365400 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:16.365402 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:16.365409 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:37:16.365528 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:16.365532 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.366649 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.366653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:16.367310 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:16.367320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.367919 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.367922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:16.368005 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:16.368077 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:37:16.368445 (+ 368us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:37:16.368452 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 02:37:16.368510 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 02:37:16.368602 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":138,"threads_started":1}
I20260812 02:37:16.368764 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:16.364718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:43279 (local address 127.3.127.130:41201)
0812 02:37:16.364879 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:16.364885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:16.365036 (+ 151us) server_negotiation.cc:407] Connection header received
0812 02:37:16.365226 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:16.365230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:16.365300 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:16.365391 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:16.365683 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.366470 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.367447 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.367784 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.367904 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:16.368235 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:16.368341 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:16.368424 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:16.368546 (+ 122us) server_negotiation.cc:299] Negotiation successful
0812 02:37:16.368618 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 02:37:16.376154 6946 raft_consensus.cc:1261] T 4569eb380a76498dab96b172aa142d72 P 8f35f8c70bc8444ebda49652410026fa [term 1 FOLLOWER]: Refusing update from remote peer 477261cc28a74a3cbfadc6e51e6a0bf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:16.376847 7176 raft_consensus.cc:1261] T 4569eb380a76498dab96b172aa142d72 P ae72c9b1260841c6af443de9be6cbbc0 [term 1 FOLLOWER]: Refusing update from remote peer 477261cc28a74a3cbfadc6e51e6a0bf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:16.377205 7370 consensus_queue.cc:1035] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:16.377584 7300 consensus_queue.cc:1035] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:16.403484 7348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:16.399468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:33869 (local address 127.3.127.130:43893)
0812 02:37:16.399656 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:16.399672 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:16.399766 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:16.400039 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:16.400042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:16.400050 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:37:16.400168 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:16.400174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.400956 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.400960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:16.401922 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:16.401935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.402493 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.402497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:16.402592 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:16.402621 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:37:16.403153 (+ 532us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:37:16.403161 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 02:37:16.403223 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 02:37:16.403332 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260812 02:37:16.403999 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:16.399536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:43893 (local address 127.3.127.129:33869)
0812 02:37:16.399665 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:16.399669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:16.399718 (+ 49us) server_negotiation.cc:407] Connection header received
0812 02:37:16.399889 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:16.399892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:16.399951 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:16.400020 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:16.400272 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.400854 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.402067 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:16.402370 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:16.402476 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:16.402980 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:16.403064 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:16.403127 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:16.403829 (+ 702us) server_negotiation.cc:299] Negotiation successful
0812 02:37:16.403897 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260812 02:37:16.405002 7333 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 02:37:17.982547 6427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:17.982801 6427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:17.985886 6427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c: Sending DeleteTablet for 3 replicas of tablet 4569eb380a76498dab96b172aa142d72
I20260812 02:37:17.986620 6676 tablet_service.cc:1514] Processing DeleteTablet for tablet 4569eb380a76498dab96b172aa142d72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:17 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:53352
I20260812 02:37:17.986678 6914 tablet_service.cc:1514] Processing DeleteTablet for tablet 4569eb380a76498dab96b172aa142d72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:17 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:32878
I20260812 02:37:17.986732 7144 tablet_service.cc:1514] Processing DeleteTablet for tablet 4569eb380a76498dab96b172aa142d72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:17 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:39206
I20260812 02:37:17.987067 7431 tablet_replica.cc:330] T 4569eb380a76498dab96b172aa142d72 P 8f35f8c70bc8444ebda49652410026fa: stopping tablet replica
I20260812 02:37:17.987094 7430 tablet_replica.cc:330] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5: stopping tablet replica
I20260812 02:37:17.987226 7430 raft_consensus.cc:2229] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:37:17.987226 7431 raft_consensus.cc:2229] T 4569eb380a76498dab96b172aa142d72 P 8f35f8c70bc8444ebda49652410026fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:17.987416 7431 raft_consensus.cc:2258] T 4569eb380a76498dab96b172aa142d72 P 8f35f8c70bc8444ebda49652410026fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:17.987416 7432 tablet_replica.cc:330] T 4569eb380a76498dab96b172aa142d72 P ae72c9b1260841c6af443de9be6cbbc0: stopping tablet replica
I20260812 02:37:17.987645 7430 raft_consensus.cc:2258] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:17.987890 7432 raft_consensus.cc:2229] T 4569eb380a76498dab96b172aa142d72 P ae72c9b1260841c6af443de9be6cbbc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:17.988066 7432 raft_consensus.cc:2258] T 4569eb380a76498dab96b172aa142d72 P ae72c9b1260841c6af443de9be6cbbc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:17.988119 7431 ts_tablet_manager.cc:1905] T 4569eb380a76498dab96b172aa142d72 P 8f35f8c70bc8444ebda49652410026fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:17.988758 7432 ts_tablet_manager.cc:1905] T 4569eb380a76498dab96b172aa142d72 P ae72c9b1260841c6af443de9be6cbbc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:17.991362 7431 ts_tablet_manager.cc:1918] T 4569eb380a76498dab96b172aa142d72 P 8f35f8c70bc8444ebda49652410026fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 02:37:17.991452 7431 log.cc:1196] T 4569eb380a76498dab96b172aa142d72 P 8f35f8c70bc8444ebda49652410026fa: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/4569eb380a76498dab96b172aa142d72
I20260812 02:37:17.991698 7431 ts_tablet_manager.cc:1939] T 4569eb380a76498dab96b172aa142d72 P 8f35f8c70bc8444ebda49652410026fa: Deleting consensus metadata
I20260812 02:37:17.992136 7432 ts_tablet_manager.cc:1918] T 4569eb380a76498dab96b172aa142d72 P ae72c9b1260841c6af443de9be6cbbc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 02:37:17.992309 7432 log.cc:1196] T 4569eb380a76498dab96b172aa142d72 P ae72c9b1260841c6af443de9be6cbbc0: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/4569eb380a76498dab96b172aa142d72
I20260812 02:37:17.992399 7430 ts_tablet_manager.cc:1905] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:17.992517 6406 catalog_manager.cc:4841] TS 8f35f8c70bc8444ebda49652410026fa (127.3.127.130:41201): tablet 4569eb380a76498dab96b172aa142d72 (table test-table [id=998c17848f3f448485c46968151d3d6d]) successfully deleted
I20260812 02:37:17.992739 7432 ts_tablet_manager.cc:1939] T 4569eb380a76498dab96b172aa142d72 P ae72c9b1260841c6af443de9be6cbbc0: Deleting consensus metadata
I20260812 02:37:17.993530 6407 catalog_manager.cc:4841] TS ae72c9b1260841c6af443de9be6cbbc0 (127.3.127.131:36923): tablet 4569eb380a76498dab96b172aa142d72 (table test-table [id=998c17848f3f448485c46968151d3d6d]) successfully deleted
I20260812 02:37:17.995198 7430 ts_tablet_manager.cc:1918] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 02:37:17.995275 7430 log.cc:1196] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-0/wal/wals/4569eb380a76498dab96b172aa142d72
I20260812 02:37:17.995535 7430 ts_tablet_manager.cc:1939] T 4569eb380a76498dab96b172aa142d72 P 477261cc28a74a3cbfadc6e51e6a0bf5: Deleting consensus metadata
I20260812 02:37:17.996282 6406 catalog_manager.cc:4841] TS 477261cc28a74a3cbfadc6e51e6a0bf5 (127.3.127.129:33869): tablet 4569eb380a76498dab96b172aa142d72 (table test-table [id=998c17848f3f448485c46968151d3d6d]) successfully deleted
Aug 12 02:37:17 dist-test-slave-rmhg krb5kdc[6354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502237, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 02:37:18 dist-test-slave-rmhg krb5kdc[6354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502237, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:18.024881 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:18.009501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46830 (local address 127.3.127.190:33073)
0812 02:37:18.009813 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:18.009820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:18.009842 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:37:18.009986 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:18.009989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:18.010041 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:18.010150 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:18.010936 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.011570 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.012532 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.012809 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.012898 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:18.021307 (+ 8409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:18.021312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:18.021314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:18.021355 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:18.022903 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:18.023412 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:18.023416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:18.023417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:18.023536 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:18.023846 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:18.023850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:18.023851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:18.024085 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:18.024211 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:18.024502 (+ 291us) server_negotiation.cc:299] Negotiation successful
0812 02:37:18.024687 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":111,"threads_started":1}
I20260812 02:37:18.027941 6427 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:37:18.028416 6427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:37:18.033808 6914 tablet_service.cc:1467] Processing CreateTablet for tablet fbb517937f0c47babf9c8235951d218d (DEFAULT_TABLE table=test-table [id=63b3c5d497a048b7a48bb01edc8bc976]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:18.033816 7144 tablet_service.cc:1467] Processing CreateTablet for tablet fbb517937f0c47babf9c8235951d218d (DEFAULT_TABLE table=test-table [id=63b3c5d497a048b7a48bb01edc8bc976]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:18.033806 6676 tablet_service.cc:1467] Processing CreateTablet for tablet fbb517937f0c47babf9c8235951d218d (DEFAULT_TABLE table=test-table [id=63b3c5d497a048b7a48bb01edc8bc976]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:18.034149 7144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbb517937f0c47babf9c8235951d218d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:18.034149 6676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbb517937f0c47babf9c8235951d218d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:18.034152 6914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbb517937f0c47babf9c8235951d218d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:18.036403 7450 tablet_bootstrap.cc:492] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa: Bootstrap starting.
I20260812 02:37:18.036865 7452 tablet_bootstrap.cc:492] T fbb517937f0c47babf9c8235951d218d P 477261cc28a74a3cbfadc6e51e6a0bf5: Bootstrap starting.
I20260812 02:37:18.037433 7450 tablet_bootstrap.cc:654] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:18.037619 7453 tablet_bootstrap.cc:492] T fbb517937f0c47babf9c8235951d218d P ae72c9b1260841c6af443de9be6cbbc0: Bootstrap starting.
I20260812 02:37:18.037855 7452 tablet_bootstrap.cc:654] T fbb517937f0c47babf9c8235951d218d P 477261cc28a74a3cbfadc6e51e6a0bf5: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:18.038774 7450 tablet_bootstrap.cc:492] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa: No bootstrap required, opened a new log
I20260812 02:37:18.038841 7453 tablet_bootstrap.cc:654] T fbb517937f0c47babf9c8235951d218d P ae72c9b1260841c6af443de9be6cbbc0: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:18.038856 7450 ts_tablet_manager.cc:1397] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:37:18.039263 7450 raft_consensus.cc:348] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:18.039418 7450 raft_consensus.cc:374] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:18.039472 7450 raft_consensus.cc:729] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f35f8c70bc8444ebda49652410026fa, State: Initialized, Role: FOLLOWER
I20260812 02:37:18.039623 7450 consensus_queue.cc:260] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:18.039924 7450 ts_tablet_manager.cc:1428] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:37:18.040064 7452 tablet_bootstrap.cc:492] T fbb517937f0c47babf9c8235951d218d P 477261cc28a74a3cbfadc6e51e6a0bf5: No bootstrap required, opened a new log
I20260812 02:37:18.040165 7452 ts_tablet_manager.cc:1397] T fbb517937f0c47babf9c8235951d218d P 477261cc28a74a3cbfadc6e51e6a0bf5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:37:18.040447 7453 tablet_bootstrap.cc:492] T fbb517937f0c47babf9c8235951d218d P ae72c9b1260841c6af443de9be6cbbc0: No bootstrap required, opened a new log
I20260812 02:37:18.040593 7453 ts_tablet_manager.cc:1397] T fbb517937f0c47babf9c8235951d218d P ae72c9b1260841c6af443de9be6cbbc0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:37:18.040596 7452 raft_consensus.cc:348] T fbb517937f0c47babf9c8235951d218d P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:18.040721 7452 raft_consensus.cc:374] T fbb517937f0c47babf9c8235951d218d P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:18.040756 7452 raft_consensus.cc:729] T fbb517937f0c47babf9c8235951d218d P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 477261cc28a74a3cbfadc6e51e6a0bf5, State: Initialized, Role: FOLLOWER
I20260812 02:37:18.040886 7452 consensus_queue.cc:260] T fbb517937f0c47babf9c8235951d218d P 477261cc28a74a3cbfadc6e51e6a0bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:18.041131 7452 ts_tablet_manager.cc:1428] T fbb517937f0c47babf9c8235951d218d P 477261cc28a74a3cbfadc6e51e6a0bf5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 02:37:18.041113 7453 raft_consensus.cc:348] T fbb517937f0c47babf9c8235951d218d P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:18.041265 7453 raft_consensus.cc:374] T fbb517937f0c47babf9c8235951d218d P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:18.041301 7453 raft_consensus.cc:729] T fbb517937f0c47babf9c8235951d218d P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae72c9b1260841c6af443de9be6cbbc0, State: Initialized, Role: FOLLOWER
I20260812 02:37:18.041452 7453 consensus_queue.cc:260] T fbb517937f0c47babf9c8235951d218d P ae72c9b1260841c6af443de9be6cbbc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:18.041832 7453 ts_tablet_manager.cc:1428] T fbb517937f0c47babf9c8235951d218d P ae72c9b1260841c6af443de9be6cbbc0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:37:18.072064 7298 raft_consensus.cc:482] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:37:18.072221 7298 raft_consensus.cc:504] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:18.072682 7298 leader_election.cc:288] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 477261cc28a74a3cbfadc6e51e6a0bf5 (127.3.127.129:33869), ae72c9b1260841c6af443de9be6cbbc0 (127.3.127.131:36923)
I20260812 02:37:18.073237 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb517937f0c47babf9c8235951d218d" candidate_uuid: "8f35f8c70bc8444ebda49652410026fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" is_pre_election: true
I20260812 02:37:18.073235 7176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb517937f0c47babf9c8235951d218d" candidate_uuid: "8f35f8c70bc8444ebda49652410026fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae72c9b1260841c6af443de9be6cbbc0" is_pre_election: true
I20260812 02:37:18.073415 7176 raft_consensus.cc:2454] T fbb517937f0c47babf9c8235951d218d P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f35f8c70bc8444ebda49652410026fa in term 0.
I20260812 02:37:18.073414 6711 raft_consensus.cc:2454] T fbb517937f0c47babf9c8235951d218d P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f35f8c70bc8444ebda49652410026fa in term 0.
I20260812 02:37:18.073796 6837 leader_election.cc:302] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 477261cc28a74a3cbfadc6e51e6a0bf5, 8f35f8c70bc8444ebda49652410026fa, ae72c9b1260841c6af443de9be6cbbc0; no voters:
I20260812 02:37:18.074010 7298 raft_consensus.cc:2781] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:37:18.074086 7298 raft_consensus.cc:482] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:37:18.074115 7298 raft_consensus.cc:3037] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:18.075124 7298 raft_consensus.cc:504] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:18.075477 7298 leader_election.cc:288] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa [CANDIDATE]: Term 1 election: Requested vote from peers 477261cc28a74a3cbfadc6e51e6a0bf5 (127.3.127.129:33869), ae72c9b1260841c6af443de9be6cbbc0 (127.3.127.131:36923)
I20260812 02:37:18.075877 7176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb517937f0c47babf9c8235951d218d" candidate_uuid: "8f35f8c70bc8444ebda49652410026fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae72c9b1260841c6af443de9be6cbbc0"
I20260812 02:37:18.075917 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb517937f0c47babf9c8235951d218d" candidate_uuid: "8f35f8c70bc8444ebda49652410026fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5"
I20260812 02:37:18.076001 7176 raft_consensus.cc:3037] T fbb517937f0c47babf9c8235951d218d P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:18.076041 6711 raft_consensus.cc:3037] T fbb517937f0c47babf9c8235951d218d P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:18.076989 7176 raft_consensus.cc:2454] T fbb517937f0c47babf9c8235951d218d P ae72c9b1260841c6af443de9be6cbbc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f35f8c70bc8444ebda49652410026fa in term 1.
I20260812 02:37:18.077072 6711 raft_consensus.cc:2454] T fbb517937f0c47babf9c8235951d218d P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f35f8c70bc8444ebda49652410026fa in term 1.
I20260812 02:37:18.077316 6838 leader_election.cc:302] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f35f8c70bc8444ebda49652410026fa, ae72c9b1260841c6af443de9be6cbbc0; no voters:
I20260812 02:37:18.077520 7298 raft_consensus.cc:2781] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:18.077626 7298 raft_consensus.cc:686] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa [term 1 LEADER]: Becoming Leader. State: Replica: 8f35f8c70bc8444ebda49652410026fa, State: Running, Role: LEADER
I20260812 02:37:18.077760 7298 consensus_queue.cc:237] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:18.079670 6426 catalog_manager.cc:5495] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f35f8c70bc8444ebda49652410026fa (127.3.127.130). New cstate: current_term: 1 leader_uuid: "8f35f8c70bc8444ebda49652410026fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } health_report { overall_health: UNKNOWN } } }
I20260812 02:37:18.103366 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:18.099370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32918 (local address 127.3.127.130:41201)
0812 02:37:18.099719 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:18.099727 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:18.099740 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:37:18.099782 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:18.099784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:18.099835 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:18.099908 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:18.100355 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.100959 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.102046 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.102320 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.102410 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:18.102854 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:18.102949 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:18.103006 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:18.103138 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 02:37:18.103196 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":168,"threads_started":1}
I20260812 02:37:18.110477 7176 raft_consensus.cc:1261] T fbb517937f0c47babf9c8235951d218d P ae72c9b1260841c6af443de9be6cbbc0 [term 1 FOLLOWER]: Refusing update from remote peer 8f35f8c70bc8444ebda49652410026fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:18.110477 6711 raft_consensus.cc:1261] T fbb517937f0c47babf9c8235951d218d P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 1 FOLLOWER]: Refusing update from remote peer 8f35f8c70bc8444ebda49652410026fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:18.110877 7407 consensus_queue.cc:1035] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:18.111038 7298 consensus_queue.cc:1035] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:18.144580 6426 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:18.144747 6426 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:18.146914 6426 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c: Sending DeleteTablet for 3 replicas of tablet fbb517937f0c47babf9c8235951d218d
I20260812 02:37:18.147508 6913 tablet_service.cc:1514] Processing DeleteTablet for tablet fbb517937f0c47babf9c8235951d218d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:18 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:32878
I20260812 02:37:18.147506 6676 tablet_service.cc:1514] Processing DeleteTablet for tablet fbb517937f0c47babf9c8235951d218d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:18 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:53352
I20260812 02:37:18.147598 7144 tablet_service.cc:1514] Processing DeleteTablet for tablet fbb517937f0c47babf9c8235951d218d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:18 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:39206
I20260812 02:37:18.147709 7430 tablet_replica.cc:330] T fbb517937f0c47babf9c8235951d218d P 477261cc28a74a3cbfadc6e51e6a0bf5: stopping tablet replica
I20260812 02:37:18.147831 7432 tablet_replica.cc:330] T fbb517937f0c47babf9c8235951d218d P ae72c9b1260841c6af443de9be6cbbc0: stopping tablet replica
I20260812 02:37:18.147826 7431 tablet_replica.cc:330] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa: stopping tablet replica
I20260812 02:37:18.147830 7430 raft_consensus.cc:2229] T fbb517937f0c47babf9c8235951d218d P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:18.147938 7432 raft_consensus.cc:2229] T fbb517937f0c47babf9c8235951d218d P ae72c9b1260841c6af443de9be6cbbc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:18.147956 7430 raft_consensus.cc:2258] T fbb517937f0c47babf9c8235951d218d P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:18.147958 7431 raft_consensus.cc:2229] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:37:18.148041 7432 raft_consensus.cc:2258] T fbb517937f0c47babf9c8235951d218d P ae72c9b1260841c6af443de9be6cbbc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:18.148169 7431 raft_consensus.cc:2258] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:18.148361 6426 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:37:18.148783 6426 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:37:18.148830 7430 ts_tablet_manager.cc:1905] T fbb517937f0c47babf9c8235951d218d P 477261cc28a74a3cbfadc6e51e6a0bf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:18.149132 7432 ts_tablet_manager.cc:1905] T fbb517937f0c47babf9c8235951d218d P ae72c9b1260841c6af443de9be6cbbc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:18.149170 7431 ts_tablet_manager.cc:1905] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:18.151885 7430 ts_tablet_manager.cc:1918] T fbb517937f0c47babf9c8235951d218d P 477261cc28a74a3cbfadc6e51e6a0bf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:37:18.151973 7430 log.cc:1196] T fbb517937f0c47babf9c8235951d218d P 477261cc28a74a3cbfadc6e51e6a0bf5: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-0/wal/wals/fbb517937f0c47babf9c8235951d218d
I20260812 02:37:18.151978 7431 ts_tablet_manager.cc:1918] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:37:18.152064 7431 log.cc:1196] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/fbb517937f0c47babf9c8235951d218d
I20260812 02:37:18.152271 7430 ts_tablet_manager.cc:1939] T fbb517937f0c47babf9c8235951d218d P 477261cc28a74a3cbfadc6e51e6a0bf5: Deleting consensus metadata
I20260812 02:37:18.152313 7431 ts_tablet_manager.cc:1939] T fbb517937f0c47babf9c8235951d218d P 8f35f8c70bc8444ebda49652410026fa: Deleting consensus metadata
I20260812 02:37:18.153079 6406 catalog_manager.cc:4841] TS 477261cc28a74a3cbfadc6e51e6a0bf5 (127.3.127.129:33869): tablet fbb517937f0c47babf9c8235951d218d (table test-table [id=63b3c5d497a048b7a48bb01edc8bc976]) successfully deleted
I20260812 02:37:18.153273 6406 catalog_manager.cc:4841] TS 8f35f8c70bc8444ebda49652410026fa (127.3.127.130:41201): tablet fbb517937f0c47babf9c8235951d218d (table test-table [id=63b3c5d497a048b7a48bb01edc8bc976]) successfully deleted
I20260812 02:37:18.153502 7432 ts_tablet_manager.cc:1918] T fbb517937f0c47babf9c8235951d218d P ae72c9b1260841c6af443de9be6cbbc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:37:18.153587 7432 log.cc:1196] T fbb517937f0c47babf9c8235951d218d P ae72c9b1260841c6af443de9be6cbbc0: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/fbb517937f0c47babf9c8235951d218d
I20260812 02:37:18.153843 7432 ts_tablet_manager.cc:1939] T fbb517937f0c47babf9c8235951d218d P ae72c9b1260841c6af443de9be6cbbc0: Deleting consensus metadata
I20260812 02:37:18.154377 7144 tablet_service.cc:1467] Processing CreateTablet for tablet 34dddbeeb1084a8fb1e01a57c9dfaec7 (DEFAULT_TABLE table=test-table [id=00b39764b0df4be2a5ecfb971df80af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:18.154376 6913 tablet_service.cc:1467] Processing CreateTablet for tablet 34dddbeeb1084a8fb1e01a57c9dfaec7 (DEFAULT_TABLE table=test-table [id=00b39764b0df4be2a5ecfb971df80af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:18.154376 6676 tablet_service.cc:1467] Processing CreateTablet for tablet 34dddbeeb1084a8fb1e01a57c9dfaec7 (DEFAULT_TABLE table=test-table [id=00b39764b0df4be2a5ecfb971df80af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:18.154641 6676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34dddbeeb1084a8fb1e01a57c9dfaec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:18.154644 6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34dddbeeb1084a8fb1e01a57c9dfaec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:18.154661 7144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34dddbeeb1084a8fb1e01a57c9dfaec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:18.155393 6407 catalog_manager.cc:4841] TS ae72c9b1260841c6af443de9be6cbbc0 (127.3.127.131:36923): tablet fbb517937f0c47babf9c8235951d218d (table test-table [id=63b3c5d497a048b7a48bb01edc8bc976]) successfully deleted
I20260812 02:37:18.156704 7453 tablet_bootstrap.cc:492] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0: Bootstrap starting.
I20260812 02:37:18.157085 7450 tablet_bootstrap.cc:492] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 8f35f8c70bc8444ebda49652410026fa: Bootstrap starting.
I20260812 02:37:18.157101 7452 tablet_bootstrap.cc:492] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 477261cc28a74a3cbfadc6e51e6a0bf5: Bootstrap starting.
I20260812 02:37:18.157768 7453 tablet_bootstrap.cc:654] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:18.157960 7452 tablet_bootstrap.cc:654] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 477261cc28a74a3cbfadc6e51e6a0bf5: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:18.158006 7450 tablet_bootstrap.cc:654] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 8f35f8c70bc8444ebda49652410026fa: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:18.159235 7450 tablet_bootstrap.cc:492] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 8f35f8c70bc8444ebda49652410026fa: No bootstrap required, opened a new log
I20260812 02:37:18.159317 7453 tablet_bootstrap.cc:492] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0: No bootstrap required, opened a new log
I20260812 02:37:18.159329 7450 ts_tablet_manager.cc:1397] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 8f35f8c70bc8444ebda49652410026fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 02:37:18.159428 7453 ts_tablet_manager.cc:1397] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260812 02:37:18.159540 7452 tablet_bootstrap.cc:492] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 477261cc28a74a3cbfadc6e51e6a0bf5: No bootstrap required, opened a new log
I20260812 02:37:18.159638 7452 ts_tablet_manager.cc:1397] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 477261cc28a74a3cbfadc6e51e6a0bf5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 02:37:18.159777 7450 raft_consensus.cc:348] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:18.159940 7450 raft_consensus.cc:374] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:18.159983 7450 raft_consensus.cc:729] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f35f8c70bc8444ebda49652410026fa, State: Initialized, Role: FOLLOWER
I20260812 02:37:18.159915 7453 raft_consensus.cc:348] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:18.160036 7453 raft_consensus.cc:374] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:18.160079 7453 raft_consensus.cc:729] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae72c9b1260841c6af443de9be6cbbc0, State: Initialized, Role: FOLLOWER
I20260812 02:37:18.160133 7450 consensus_queue.cc:260] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 8f35f8c70bc8444ebda49652410026fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:18.160125 7452 raft_consensus.cc:348] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:18.160228 7452 raft_consensus.cc:374] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:18.160264 7452 raft_consensus.cc:729] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 477261cc28a74a3cbfadc6e51e6a0bf5, State: Initialized, Role: FOLLOWER
I20260812 02:37:18.160254 7453 consensus_queue.cc:260] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:18.160421 7450 ts_tablet_manager.cc:1428] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 8f35f8c70bc8444ebda49652410026fa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:37:18.160396 7452 consensus_queue.cc:260] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 477261cc28a74a3cbfadc6e51e6a0bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:18.160531 7453 ts_tablet_manager.cc:1428] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260812 02:37:18.160661 7452 ts_tablet_manager.cc:1428] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 477261cc28a74a3cbfadc6e51e6a0bf5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:37:18.296545 7460 raft_consensus.cc:482] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:37:18.296741 7460 raft_consensus.cc:504] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:18.298200 7460 leader_election.cc:288] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 477261cc28a74a3cbfadc6e51e6a0bf5 (127.3.127.129:33869), 8f35f8c70bc8444ebda49652410026fa (127.3.127.130:41201)
I20260812 02:37:18.302881 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:18.299148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:34609 (local address 127.3.127.130:41201)
0812 02:37:18.299283 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:18.299287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:18.299424 (+ 137us) server_negotiation.cc:407] Connection header received
0812 02:37:18.299547 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:18.299591 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:18.299636 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:18.299701 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:18.299973 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.300483 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.301748 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.302183 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.302334 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:18.302395 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:18.302662 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 02:37:18.302746 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 02:37:18.302915 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:18.298881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:41201 (local address 127.3.127.131:34609)
0812 02:37:18.299340 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:18.299351 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:18.299449 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:18.299734 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:18.299737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:18.299753 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:18.299873 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:18.299878 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.300611 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.300615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:18.301621 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:18.301632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.302437 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.302441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:18.302531 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:18.302554 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 02:37:18.302625 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 02:37:18.302708 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":48,"thread_start_us":262,"threads_started":1}
I20260812 02:37:18.302922 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:18.298633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:49569 (local address 127.3.127.129:33869)
0812 02:37:18.298957 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:18.298975 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:18.298994 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:37:18.299065 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:18.299068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:18.299138 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:18.299239 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:18.299702 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.300255 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.301757 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.302194 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.302319 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:18.302383 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:18.302696 (+ 313us) server_negotiation.cc:299] Negotiation successful
0812 02:37:18.302792 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":112,"threads_started":1}
I20260812 02:37:18.302970 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:18.298508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:33869 (local address 127.3.127.131:49569)
0812 02:37:18.298804 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:18.298822 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:18.298972 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:18.299336 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:18.299340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:18.299361 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:18.299573 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:18.299580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.300375 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.300378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:18.301620 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:18.301633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.302395 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.302398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:18.302562 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:18.302583 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 02:37:18.302656 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 02:37:18.302739 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":9,"thread_start_us":82,"threads_started":1}
I20260812 02:37:18.303479 6946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34dddbeeb1084a8fb1e01a57c9dfaec7" candidate_uuid: "ae72c9b1260841c6af443de9be6cbbc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f35f8c70bc8444ebda49652410026fa" is_pre_election: true
I20260812 02:37:18.303619 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34dddbeeb1084a8fb1e01a57c9dfaec7" candidate_uuid: "ae72c9b1260841c6af443de9be6cbbc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" is_pre_election: true
I20260812 02:37:18.303727 6946 raft_consensus.cc:2454] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae72c9b1260841c6af443de9be6cbbc0 in term 0.
I20260812 02:37:18.303788 6711 raft_consensus.cc:2454] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae72c9b1260841c6af443de9be6cbbc0 in term 0.
I20260812 02:37:18.304201 7065 leader_election.cc:302] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 477261cc28a74a3cbfadc6e51e6a0bf5, ae72c9b1260841c6af443de9be6cbbc0; no voters:
I20260812 02:37:18.304518 7460 raft_consensus.cc:2781] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:37:18.304587 7460 raft_consensus.cc:482] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:37:18.304622 7460 raft_consensus.cc:3037] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:18.306336 7460 raft_consensus.cc:504] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:18.306707 7460 leader_election.cc:288] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 [CANDIDATE]: Term 1 election: Requested vote from peers 477261cc28a74a3cbfadc6e51e6a0bf5 (127.3.127.129:33869), 8f35f8c70bc8444ebda49652410026fa (127.3.127.130:41201)
I20260812 02:37:18.307207 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34dddbeeb1084a8fb1e01a57c9dfaec7" candidate_uuid: "ae72c9b1260841c6af443de9be6cbbc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5"
I20260812 02:37:18.307256 6946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34dddbeeb1084a8fb1e01a57c9dfaec7" candidate_uuid: "ae72c9b1260841c6af443de9be6cbbc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f35f8c70bc8444ebda49652410026fa"
I20260812 02:37:18.307365 6711 raft_consensus.cc:3037] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:18.307389 6946 raft_consensus.cc:3037] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 8f35f8c70bc8444ebda49652410026fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:18.308637 6711 raft_consensus.cc:2454] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae72c9b1260841c6af443de9be6cbbc0 in term 1.
I20260812 02:37:18.308638 6946 raft_consensus.cc:2454] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 8f35f8c70bc8444ebda49652410026fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae72c9b1260841c6af443de9be6cbbc0 in term 1.
I20260812 02:37:18.309024 7065 leader_election.cc:302] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f35f8c70bc8444ebda49652410026fa, ae72c9b1260841c6af443de9be6cbbc0; no voters:
I20260812 02:37:18.309374 7460 raft_consensus.cc:2781] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:18.309645 7460 raft_consensus.cc:686] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 [term 1 LEADER]: Becoming Leader. State: Replica: ae72c9b1260841c6af443de9be6cbbc0, State: Running, Role: LEADER
I20260812 02:37:18.309916 7460 consensus_queue.cc:237] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } }
I20260812 02:37:18.312633 6426 catalog_manager.cc:5495] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae72c9b1260841c6af443de9be6cbbc0 (127.3.127.131). New cstate: current_term: 1 leader_uuid: "ae72c9b1260841c6af443de9be6cbbc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae72c9b1260841c6af443de9be6cbbc0" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36923 } health_report { overall_health: HEALTHY } } }
I20260812 02:37:18.365967 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:18.361339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39216 (local address 127.3.127.131:36923)
0812 02:37:18.361661 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:18.361669 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:18.361683 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:37:18.361767 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:18.361769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:18.361828 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:18.361942 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:18.362197 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.362797 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.364235 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.364535 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.364649 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:18.364943 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:18.365513 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:18.365664 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:18.365753 (+ 89us) server_negotiation.cc:299] Negotiation successful
0812 02:37:18.365821 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":147,"threads_started":1}
I20260812 02:37:18.379093 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:18.374768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:45335 (local address 127.3.127.130:41201)
0812 02:37:18.374947 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:18.374952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:18.375084 (+ 132us) server_negotiation.cc:407] Connection header received
0812 02:37:18.375313 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:18.375316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:18.375384 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:18.375471 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:18.375964 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.376494 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.377625 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.377875 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.377974 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:18.378583 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:18.378688 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:18.378752 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:18.378856 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 02:37:18.378922 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 02:37:18.379487 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:18.374596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:41201 (local address 127.3.127.131:45335)
0812 02:37:18.374974 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:18.374991 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:18.375157 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:18.375513 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:18.375516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:18.375525 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:37:18.375830 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:18.375840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.376613 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.376615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:18.377491 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:18.377505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.378137 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.378151 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:18.378322 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:18.378400 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:37:18.378775 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:37:18.378783 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 02:37:18.378846 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 02:37:18.378951 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":138,"threads_started":1}
I20260812 02:37:18.386289 6946 raft_consensus.cc:1261] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 8f35f8c70bc8444ebda49652410026fa [term 1 FOLLOWER]: Refusing update from remote peer ae72c9b1260841c6af443de9be6cbbc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:18.386323 6711 raft_consensus.cc:1261] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 477261cc28a74a3cbfadc6e51e6a0bf5 [term 1 FOLLOWER]: Refusing update from remote peer ae72c9b1260841c6af443de9be6cbbc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:18.386763 7460 consensus_queue.cc:1035] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f35f8c70bc8444ebda49652410026fa" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 41201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:18.387130 7475 consensus_queue.cc:1035] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "477261cc28a74a3cbfadc6e51e6a0bf5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:18.406303 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:18.402221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:53511 (local address 127.3.127.131:36923)
0812 02:37:18.402340 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:18.402345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:18.402804 (+ 459us) server_negotiation.cc:407] Connection header received
0812 02:37:18.403094 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:18.403097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:18.403158 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:18.403232 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:18.403454 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.404025 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.405204 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.405511 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.405619 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:18.405909 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:18.406004 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:18.406063 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:18.406135 (+ 72us) server_negotiation.cc:299] Negotiation successful
0812 02:37:18.406189 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 02:37:18.406337 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:18.402158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:36923 (local address 127.3.127.130:53511)
0812 02:37:18.402651 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:18.402667 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:18.402837 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:18.403263 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:18.403265 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:18.403274 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:37:18.403381 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:18.403385 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.404116 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.404119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:18.405068 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:18.405080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:18.405631 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:18.405635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:18.405749 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:18.405790 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:37:18.406072 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:37:18.406078 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 02:37:18.406126 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 02:37:18.406213 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":225,"threads_started":1}
W20260812 02:37:18.407140 7401 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 02:37:19.970315 6427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:19.970479 6427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:19.973306 6427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dee11b974a3d4738b8f9a6170354282c: Sending DeleteTablet for 3 replicas of tablet 34dddbeeb1084a8fb1e01a57c9dfaec7
I20260812 02:37:19.974015 7144 tablet_service.cc:1514] Processing DeleteTablet for tablet 34dddbeeb1084a8fb1e01a57c9dfaec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:19 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:39206
I20260812 02:37:19.974015 6676 tablet_service.cc:1514] Processing DeleteTablet for tablet 34dddbeeb1084a8fb1e01a57c9dfaec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:19 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:53352
I20260812 02:37:19.974056 6913 tablet_service.cc:1514] Processing DeleteTablet for tablet 34dddbeeb1084a8fb1e01a57c9dfaec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:19 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:32878
I20260812 02:37:19.974507 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 6560
I20260812 02:37:19.974550 7545 tablet_replica.cc:330] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 8f35f8c70bc8444ebda49652410026fa: stopping tablet replica
I20260812 02:37:20.011518 7545 raft_consensus.cc:2229] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 8f35f8c70bc8444ebda49652410026fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:20.011714 7545 raft_consensus.cc:2258] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 8f35f8c70bc8444ebda49652410026fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:20.016330 7545 ts_tablet_manager.cc:1905] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 8f35f8c70bc8444ebda49652410026fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:20.019903 7545 ts_tablet_manager.cc:1918] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 8f35f8c70bc8444ebda49652410026fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 02:37:20.020032 7545 log.cc:1196] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 8f35f8c70bc8444ebda49652410026fa: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/34dddbeeb1084a8fb1e01a57c9dfaec7
I20260812 02:37:20.020352 7545 ts_tablet_manager.cc:1939] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P 8f35f8c70bc8444ebda49652410026fa: Deleting consensus metadata
I20260812 02:37:20.020757 7544 tablet_replica.cc:330] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0: stopping tablet replica
I20260812 02:37:20.020915 7544 raft_consensus.cc:2229] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:37:20.021306 7544 raft_consensus.cc:2258] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:20.022739 7544 ts_tablet_manager.cc:1905] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:20.026532 7544 ts_tablet_manager.cc:1918] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 02:37:20.026648 7544 log.cc:1196] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/34dddbeeb1084a8fb1e01a57c9dfaec7
I20260812 02:37:20.027002 7544 ts_tablet_manager.cc:1939] T 34dddbeeb1084a8fb1e01a57c9dfaec7 P ae72c9b1260841c6af443de9be6cbbc0: Deleting consensus metadata
I20260812 02:37:20.028081 6407 catalog_manager.cc:4841] TS ae72c9b1260841c6af443de9be6cbbc0 (127.3.127.131:36923): tablet 34dddbeeb1084a8fb1e01a57c9dfaec7 (table test-table [id=00b39764b0df4be2a5ecfb971df80af8]) successfully deleted
I20260812 02:37:20.033135 6406 catalog_manager.cc:4841] TS 8f35f8c70bc8444ebda49652410026fa (127.3.127.130:41201): tablet 34dddbeeb1084a8fb1e01a57c9dfaec7 (table test-table [id=00b39764b0df4be2a5ecfb971df80af8]) successfully deleted
W20260812 02:37:20.040458 6406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.127.129:33869 (error 108)
I20260812 02:37:20.041584 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 6810
I20260812 02:37:20.048761 7548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:20.041412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:33869 (local address 127.0.0.1:60590)
0812 02:37:20.048284 (+ 6872us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:20.048546 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.129:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":98,"threads_started":1}
W20260812 02:37:20.055070 6406 catalog_manager.cc:4584] TS 477261cc28a74a3cbfadc6e51e6a0bf5 (127.3.127.129:33869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 34dddbeeb1084a8fb1e01a57c9dfaec7: Network error: Client connection negotiation failed: client connection to 127.3.127.129:33869: connect: Connection refused (error 111)
I20260812 02:37:20.066442 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 7040
I20260812 02:37:20.089414 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 6375
2026-08-12T02:37:20Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6710 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:37:20 dist-test-slave-rmhg krb5kdc[7552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:37:20 dist-test-slave-rmhg krb5kdc[7552](info): set up 2 sockets
Aug 12 02:37:20 dist-test-slave-rmhg krb5kdc[7552](info): commencing operation
krb5kdc: starting...
W20260812 02:37:21.273728 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:37:21 dist-test-slave-rmhg krb5kdc[7552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:37:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:37:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:37:21.408761 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:37887
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:37887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:21.532733 7577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:21.533041 7577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:21.533082 7577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:21.537472 7577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:21.537544 7577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:21.537562 7577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:21.537577 7577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:21.537592 7577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:37:21.543138 7577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:37887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:37887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:21.544396 7577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:21.545554 7577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:21.555058 7586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:21.555063 7590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:21.555063 7594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:21.555536 7577 server_base.cc:1034] running on GCE node
I20260812 02:37:21.556206 7577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:21.557497 7577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:21.558702 7577 hybrid_clock.cc:648] HybridClock initialized: now 1786502241558669 us; error 46 us; skew 500 ppm
Aug 12 02:37:21 dist-test-slave-rmhg krb5kdc[7552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502241, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:21.561756 7577 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:37:21.562777 7577 webserver.cc:460] Webserver started at http://127.3.127.190:34789/ using document root <none> and password file <none>
I20260812 02:37:21.563385 7577 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:21.563458 7577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:21.563695 7577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:21.566011 7577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "b4e89404c4794fac920152fdcaf5a874"
format_stamp: "Formatted at 2026-08-12 02:37:21 on dist-test-slave-rmhg"
server_key: "905e4ceb2e7f96cf752d818f0a481215"
server_key_iv: "9ba469b121045618051c3ca917d4cec0"
server_key_version: "encryptionkey@0"
I20260812 02:37:21.566602 7577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "b4e89404c4794fac920152fdcaf5a874"
format_stamp: "Formatted at 2026-08-12 02:37:21 on dist-test-slave-rmhg"
server_key: "905e4ceb2e7f96cf752d818f0a481215"
server_key_iv: "9ba469b121045618051c3ca917d4cec0"
server_key_version: "encryptionkey@0"
I20260812 02:37:21.570748 7577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:37:21.573515 7605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:21.574993 7577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 02:37:21.575167 7577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "b4e89404c4794fac920152fdcaf5a874"
format_stamp: "Formatted at 2026-08-12 02:37:21 on dist-test-slave-rmhg"
server_key: "905e4ceb2e7f96cf752d818f0a481215"
server_key_iv: "9ba469b121045618051c3ca917d4cec0"
server_key_version: "encryptionkey@0"
I20260812 02:37:21.575294 7577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:21.600744 7577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:21.604223 7577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:21.604432 7577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:21.612056 7577 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:37887
I20260812 02:37:21.612099 7687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:37887 every 8 connection(s)
I20260812 02:37:21.613281 7577 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:37:21.616674 7688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:21.616709 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 7577
I20260812 02:37:21.616819 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:37:21.617111 3582 external_mini_cluster.cc:1428] Setting key ba7466c10455bce55f07aba52062383f
I20260812 02:37:21.622720 7688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874: Bootstrap starting.
I20260812 02:37:21.625237 7688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:21.626125 7688 log.cc:824] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:37:21 dist-test-slave-rmhg krb5kdc[7552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:21.628780 7688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874: No bootstrap required, opened a new log
I20260812 02:37:21.632706 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:21.618536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58822 (local address 127.3.127.190:37887)
0812 02:37:21.618976 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:21.618997 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:21.619199 (+ 202us) server_negotiation.cc:407] Connection header received
0812 02:37:21.619992 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:21.620012 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:21.620363 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:21.620690 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:21.621554 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:21.622584 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:21.623483 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:21.623981 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:21.624134 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:21.627153 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:21.627161 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:21.627181 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:21.627212 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:21.629096 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:21.629615 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:21.629622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:21.629628 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:21.629795 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:21.630139 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:21.630142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:21.630143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:21.630724 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:21.630941 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:21.631263 (+ 322us) server_negotiation.cc:299] Negotiation successful
0812 02:37:21.631612 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":116,"threads_started":1}
I20260812 02:37:21.633503 7688 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e89404c4794fac920152fdcaf5a874" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37887 } }
I20260812 02:37:21.633803 7688 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:21.633878 7688 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e89404c4794fac920152fdcaf5a874, State: Initialized, Role: FOLLOWER
I20260812 02:37:21.634394 7688 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e89404c4794fac920152fdcaf5a874" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37887 } }
I20260812 02:37:21.634562 7688 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:37:21.634629 7688 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:37:21.634727 7688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:21.635810 7688 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e89404c4794fac920152fdcaf5a874" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37887 } }
I20260812 02:37:21.636127 7688 leader_election.cc:302] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4e89404c4794fac920152fdcaf5a874; no voters:
I20260812 02:37:21.636413 7688 leader_election.cc:288] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:37:21.636544 7700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:21.636791 7700 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874 [term 1 LEADER]: Becoming Leader. State: Replica: b4e89404c4794fac920152fdcaf5a874, State: Running, Role: LEADER
I20260812 02:37:21.637130 7700 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e89404c4794fac920152fdcaf5a874" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37887 } }
I20260812 02:37:21.637687 7688 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:37:21.638823 7705 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4e89404c4794fac920152fdcaf5a874. Latest consensus state: current_term: 1 leader_uuid: "b4e89404c4794fac920152fdcaf5a874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e89404c4794fac920152fdcaf5a874" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37887 } } }
I20260812 02:37:21.638963 7705 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:21.639251 7702 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4e89404c4794fac920152fdcaf5a874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e89404c4794fac920152fdcaf5a874" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37887 } } }
I20260812 02:37:21.639353 7702 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:21.639472 7717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:37:21.642730 7717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:37:21.648947 7717 catalog_manager.cc:1347] Generated new cluster ID: 7a5205ecc8664ff58d5de92fd661087a
I20260812 02:37:21.649029 7717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:37:21.664055 7717 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:37:21.665509 7717 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:37:21.676400 7717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4e89404c4794fac920152fdcaf5a874: Generated new TSK 0
I20260812 02:37:21.677342 7717 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
I20260812 02:37:21.718209 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37887
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:21.843209 7739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:21.843509 7739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:21.843554 7739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:21.847828 7739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:21.847904 7739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:21.847963 7739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:37:21.853520 7739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:21.854741 7739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:21.855947 7739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:21.866817 7759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:21.866823 7750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:21.866823 7755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:21.867093 7739 server_base.cc:1034] running on GCE node
I20260812 02:37:21.867601 7739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:21.868330 7739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:21.869556 7739 hybrid_clock.cc:648] HybridClock initialized: now 1786502241869521 us; error 48 us; skew 500 ppm
Aug 12 02:37:21 dist-test-slave-rmhg krb5kdc[7552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502241, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:21.872592 7739 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:37:21.873678 7739 webserver.cc:460] Webserver started at http://127.3.127.129:46379/ using document root <none> and password file <none>
I20260812 02:37:21.874285 7739 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:21.874354 7739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:21.874581 7739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:21.876919 7739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "3c005294e88b4e2a8ab16b7915adb44b"
format_stamp: "Formatted at 2026-08-12 02:37:21 on dist-test-slave-rmhg"
server_key: "a00e2aee335740cbebb39f2b71b69847"
server_key_iv: "54d72d3276a841c7854449071a4c0ed1"
server_key_version: "encryptionkey@0"
I20260812 02:37:21.877611 7739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "3c005294e88b4e2a8ab16b7915adb44b"
format_stamp: "Formatted at 2026-08-12 02:37:21 on dist-test-slave-rmhg"
server_key: "a00e2aee335740cbebb39f2b71b69847"
server_key_iv: "54d72d3276a841c7854449071a4c0ed1"
server_key_version: "encryptionkey@0"
I20260812 02:37:21.881991 7739 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:37:21.884977 7771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:21.886714 7739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:21.886873 7739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "3c005294e88b4e2a8ab16b7915adb44b"
format_stamp: "Formatted at 2026-08-12 02:37:21 on dist-test-slave-rmhg"
server_key: "a00e2aee335740cbebb39f2b71b69847"
server_key_iv: "54d72d3276a841c7854449071a4c0ed1"
server_key_version: "encryptionkey@0"
I20260812 02:37:21.887009 7739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:21.907511 7739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:21.910881 7739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:21.911077 7739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:21.911727 7739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:21.912647 7739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:21.912699 7739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:21.912755 7739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:21.912784 7739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:21.921988 7739 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:45221
I20260812 02:37:21.922027 7949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:45221 every 8 connection(s)
I20260812 02:37:21.923032 7739 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:37:21.927052 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 7739
I20260812 02:37:21.927183 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:37:21.927476 3582 external_mini_cluster.cc:1428] Setting key 8a2400c4197d6ae1c199b5015b9cb26d
Aug 12 02:37:21 dist-test-slave-rmhg krb5kdc[7552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502241, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.127.130@KRBTEST.COM; defaulting to no policy
I20260812 02:37:21.938795 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:21.925104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:37771 (local address 127.3.127.190:37887)
0812 02:37:21.925268 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:21.925273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:21.926162 (+ 889us) server_negotiation.cc:407] Connection header received
0812 02:37:21.927910 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:21.927914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:21.927985 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:21.928075 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:21.929687 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:21.930216 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:21.931692 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:21.932038 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:21.932146 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:21.934949 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:21.934954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:21.934957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:21.934991 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:21.936499 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:21.937028 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:21.937032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:21.937034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:21.937129 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:21.937531 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:21.937535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:21.937537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:21.937786 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:21.937892 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:21.938464 (+ 572us) server_negotiation.cc:299] Negotiation successful
0812 02:37:21.938607 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 02:37:21.939819 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:21.925531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:37887 (local address 127.3.127.129:37771)
0812 02:37:21.926002 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:21.926052 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:21.927601 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:21.928277 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:21.928287 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:21.928727 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:21.929496 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:21.929531 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:21.930350 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:21.930354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:21.931552 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:21.931562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:21.932177 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:21.932180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:21.932327 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:21.932940 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:21.932957 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:21.934759 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:21.936629 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:21.936634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:21.936647 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:21.936911 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:21.937276 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:21.937279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:21.937280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:21.937424 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:21.937886 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:21.937891 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:21.938177 (+ 286us) client_negotiation.cc:769] Sending connection context
0812 02:37:21.938387 (+ 210us) client_negotiation.cc:240] Negotiation successful
0812 02:37:21.938721 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":117,"threads_started":1}
I20260812 02:37:21.941121 7951 heartbeater.cc:347] Connected to a master server at 127.3.127.190:37887
I20260812 02:37:21.941496 7951 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.130@KRBTEST.COM" created.
I20260812 02:37:21.942196 7951 heartbeater.cc:511] Master 127.3.127.190:37887 requested a full tablet report, sending...
I20260812 02:37:21.944157 7633 ts_manager.cc:194] Registered new tserver with Master: 3c005294e88b4e2a8ab16b7915adb44b (127.3.127.129:45221)
I20260812 02:37:21.945683 7633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:37771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
I20260812 02:37:21.959619 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37887
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:22.082445 7958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:22.082751 7958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:22.082798 7958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:22.087711 7958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:22.087801 7958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:22.087867 7958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:37:22.093899 7958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:22.095053 7958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:22.096095 7958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:22.106652 7968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:22.106597 7980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:22.106817 7958 server_base.cc:1034] running on GCE node
W20260812 02:37:22.106622 7975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:22.107847 7958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:22.108644 7958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:22.109850 7958 hybrid_clock.cc:648] HybridClock initialized: now 1786502242109826 us; error 38 us; skew 500 ppm
Aug 12 02:37:22 dist-test-slave-rmhg krb5kdc[7552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502242, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:22.113224 7958 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:37:22.114357 7958 webserver.cc:460] Webserver started at http://127.3.127.130:43935/ using document root <none> and password file <none>
I20260812 02:37:22.115001 7958 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:22.115092 7958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:22.115327 7958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:22.117748 7958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "9e4864e85b6144ff8ef57fab420fa8e3"
format_stamp: "Formatted at 2026-08-12 02:37:22 on dist-test-slave-rmhg"
server_key: "7b4fd165315e6208c2c393189ffbd14f"
server_key_iv: "d2201a8f8e8305dda501782e456aaa0f"
server_key_version: "encryptionkey@0"
I20260812 02:37:22.118412 7958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "9e4864e85b6144ff8ef57fab420fa8e3"
format_stamp: "Formatted at 2026-08-12 02:37:22 on dist-test-slave-rmhg"
server_key: "7b4fd165315e6208c2c393189ffbd14f"
server_key_iv: "d2201a8f8e8305dda501782e456aaa0f"
server_key_version: "encryptionkey@0"
I20260812 02:37:22.122674 7958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:37:22.125443 7991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:22.126890 7958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:22.127043 7958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "9e4864e85b6144ff8ef57fab420fa8e3"
format_stamp: "Formatted at 2026-08-12 02:37:22 on dist-test-slave-rmhg"
server_key: "7b4fd165315e6208c2c393189ffbd14f"
server_key_iv: "d2201a8f8e8305dda501782e456aaa0f"
server_key_version: "encryptionkey@0"
I20260812 02:37:22.127285 7958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:22.149819 7958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:22.153417 7958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:22.153630 7958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:22.154302 7958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:22.155212 7958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:22.155263 7958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:22.155319 7958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:22.155349 7958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:22.165143 7958 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:34667
I20260812 02:37:22.165303 8184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:34667 every 8 connection(s)
I20260812 02:37:22.166196 7958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:37:22.168128 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 7958
I20260812 02:37:22.168228 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:37:22.168541 3582 external_mini_cluster.cc:1428] Setting key 5165fb4f1b744822e8e9b932b5d1fb65
Aug 12 02:37:22 dist-test-slave-rmhg krb5kdc[7552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502242, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.127.131@KRBTEST.COM; defaulting to no policy
I20260812 02:37:22.182897 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:22.168317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:51993 (local address 127.3.127.190:37887)
0812 02:37:22.168475 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:22.168479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:22.169574 (+ 1095us) server_negotiation.cc:407] Connection header received
0812 02:37:22.171306 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:22.171310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:22.171360 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:22.171435 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:22.173068 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:22.173876 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:22.175533 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:22.175767 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:22.175851 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:22.178988 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:22.178992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:22.178994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:22.179023 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:22.180482 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:22.181185 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:22.181190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:22.181192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:22.181289 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:22.181654 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:22.181659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:22.181661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:22.181901 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:22.181988 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:22.182613 (+ 625us) server_negotiation.cc:299] Negotiation successful
0812 02:37:22.182742 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.131@KRBTEST.COM" created.
I20260812 02:37:22.184092 8188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:22.168753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:37887 (local address 127.3.127.130:51993)
0812 02:37:22.169404 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:22.169472 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:22.171012 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:22.171665 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:22.171675 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:22.172203 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:22.172891 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:22.172907 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:22.174025 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:22.174028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:22.175399 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:22.175412 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:22.175940 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:22.175945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:22.176173 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:22.176847 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:22.176871 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:22.178784 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:22.180610 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:22.180618 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:22.180637 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:22.181020 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:22.181378 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:22.181381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:22.181382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:22.181531 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:22.182033 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:22.182040 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:22.182327 (+ 287us) client_negotiation.cc:769] Sending connection context
0812 02:37:22.182570 (+ 243us) client_negotiation.cc:240] Negotiation successful
0812 02:37:22.182949 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":178,"threads_started":1}
I20260812 02:37:22.185427 8185 heartbeater.cc:347] Connected to a master server at 127.3.127.190:37887
I20260812 02:37:22.185724 8185 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:22.186260 8185 heartbeater.cc:511] Master 127.3.127.190:37887 requested a full tablet report, sending...
I20260812 02:37:22.187466 7633 ts_manager.cc:194] Registered new tserver with Master: 9e4864e85b6144ff8ef57fab420fa8e3 (127.3.127.130:34667)
I20260812 02:37:22.188225 7633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:51993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
I20260812 02:37:22.201107 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37887
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:22.323402 8193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:22.323734 8193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:22.323781 8193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:22.328025 8193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:22.328114 8193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:22.328177 8193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:37:22.334167 8193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:22.335395 8193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:22.336514 8193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:22.347223 8206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:22.347401 8193 server_base.cc:1034] running on GCE node
W20260812 02:37:22.347218 8209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:22.347241 8207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:22.347990 8193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:22.348807 8193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:22.349973 8193 hybrid_clock.cc:648] HybridClock initialized: now 1786502242349940 us; error 54 us; skew 500 ppm
Aug 12 02:37:22 dist-test-slave-rmhg krb5kdc[7552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502242, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:22.352952 8193 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:37:22.353981 8193 webserver.cc:460] Webserver started at http://127.3.127.131:35257/ using document root <none> and password file <none>
I20260812 02:37:22.354588 8193 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:22.354660 8193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:22.354874 8193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:22.357244 8193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "7541ba0baf34419888ff6c1b56afc57e"
format_stamp: "Formatted at 2026-08-12 02:37:22 on dist-test-slave-rmhg"
server_key: "f8375ffd2725c3e936e02bfc17ee1641"
server_key_iv: "e4a5cc0a7d144939ec37df6c54eff041"
server_key_version: "encryptionkey@0"
I20260812 02:37:22.357903 8193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "7541ba0baf34419888ff6c1b56afc57e"
format_stamp: "Formatted at 2026-08-12 02:37:22 on dist-test-slave-rmhg"
server_key: "f8375ffd2725c3e936e02bfc17ee1641"
server_key_iv: "e4a5cc0a7d144939ec37df6c54eff041"
server_key_version: "encryptionkey@0"
I20260812 02:37:22.362323 8193 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:37:22.365335 8221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:22.366786 8193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:37:22.366920 8193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "7541ba0baf34419888ff6c1b56afc57e"
format_stamp: "Formatted at 2026-08-12 02:37:22 on dist-test-slave-rmhg"
server_key: "f8375ffd2725c3e936e02bfc17ee1641"
server_key_iv: "e4a5cc0a7d144939ec37df6c54eff041"
server_key_version: "encryptionkey@0"
I20260812 02:37:22.367038 8193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:22.400818 8193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:22.404569 8193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:22.404796 8193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:22.405541 8193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:22.406482 8193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:22.406535 8193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:22.406591 8193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:22.406620 8193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:22.416241 8193 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:41245
I20260812 02:37:22.416275 8400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:41245 every 8 connection(s)
I20260812 02:37:22.417316 8193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:37:22.420028 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 8193
I20260812 02:37:22.420179 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:37:22.420516 3582 external_mini_cluster.cc:1428] Setting key d21d75d70d0fe9c31cca01d63dc43c6b
Aug 12 02:37:22 dist-test-slave-rmhg krb5kdc[7552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502242, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:22.432397 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:22.419383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:48239 (local address 127.3.127.190:37887)
0812 02:37:22.419545 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:22.419548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:22.420755 (+ 1207us) server_negotiation.cc:407] Connection header received
0812 02:37:22.422390 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:22.422393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:22.422452 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:22.422530 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:22.423816 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:22.424440 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:22.425723 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:22.425946 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:22.426045 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:22.428566 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:22.428570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:22.428571 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:22.428600 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:22.430085 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:22.430672 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:22.430676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:22.430677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:22.430769 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:22.431165 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:22.431168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:22.431170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:22.431368 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:22.431460 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:22.432080 (+ 620us) server_negotiation.cc:299] Negotiation successful
0812 02:37:22.432224 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 02:37:22.433323 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:22.419882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:37887 (local address 127.3.127.131:48239)
0812 02:37:22.420553 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:22.420626 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:22.422165 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:22.422739 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:22.422748 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:22.423135 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:22.423671 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:22.423686 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:22.424584 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:22.424588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:22.425598 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:22.425609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:22.426104 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:22.426108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:22.426290 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:22.426885 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:22.426903 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:22.428384 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:22.430219 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:22.430226 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:22.430242 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:22.430567 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:22.430858 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:22.430861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:22.430863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:22.431071 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:22.431523 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:22.431531 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:22.431805 (+ 274us) client_negotiation.cc:769] Sending connection context
0812 02:37:22.431991 (+ 186us) client_negotiation.cc:240] Negotiation successful
0812 02:37:22.432362 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":194,"threads_started":1}
I20260812 02:37:22.434634 8401 heartbeater.cc:347] Connected to a master server at 127.3.127.190:37887
I20260812 02:37:22.434909 8401 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:22.435441 8401 heartbeater.cc:511] Master 127.3.127.190:37887 requested a full tablet report, sending...
I20260812 02:37:22.436527 7633 ts_manager.cc:194] Registered new tserver with Master: 7541ba0baf34419888ff6c1b56afc57e (127.3.127.131:41245)
I20260812 02:37:22.437083 7633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:48239
I20260812 02:37:22.447018 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 02:37:22.462750 7697 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:22.461243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58838 (local address 127.3.127.190:37887)
0812 02:37:22.461377 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:22.461381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:22.461535 (+ 154us) server_negotiation.cc:407] Connection header received
0812 02:37:22.461677 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:22.461679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:22.461725 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:22.461807 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:22.462613 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58838: BlockingRecv error: recv got EOF from 127.0.0.1:58838 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 02:37:22.464241 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 7739
I20260812 02:37:22.512557 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 7958
I20260812 02:37:22.555439 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 8193
I20260812 02:37:22.574791 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 7577
2026-08-12T02:37:22Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2487 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:37:22 dist-test-slave-rmhg krb5kdc[8414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:37:22 dist-test-slave-rmhg krb5kdc[8414](info): set up 2 sockets
Aug 12 02:37:22 dist-test-slave-rmhg krb5kdc[8414](info): commencing operation
krb5kdc: starting...
W20260812 02:37:23.773597 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:37:23 dist-test-slave-rmhg krb5kdc[8414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:37:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:37:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:37:23.910310 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:40337
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:40337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:24.030308 8431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:24.030633 8431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:24.030678 8431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:24.034919 8431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:24.034994 8431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:24.035013 8431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:24.035028 8431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:24.035048 8431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:37:24.040640 8431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:40337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:40337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:24.041873 8431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:24.042959 8431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:24.052155 8439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:24.052217 8431 server_base.cc:1034] running on GCE node
W20260812 02:37:24.052165 8441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:24.052178 8445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:24.053061 8431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:24.054369 8431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:24.055576 8431 hybrid_clock.cc:648] HybridClock initialized: now 1786502244055570 us; error 53 us; skew 500 ppm
Aug 12 02:37:24 dist-test-slave-rmhg krb5kdc[8414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502244, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:24.063097 8431 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:37:24.064193 8431 webserver.cc:460] Webserver started at http://127.3.127.190:35321/ using document root <none> and password file <none>
I20260812 02:37:24.064805 8431 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:24.064874 8431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:24.065064 8431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:24.067382 8431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "082f2431059a4f998a7068a3abe5b890"
format_stamp: "Formatted at 2026-08-12 02:37:24 on dist-test-slave-rmhg"
server_key: "3536222e6a06ae41a3ce0994a0049477"
server_key_iv: "103710d6cde736fd3de19125fa33b3d1"
server_key_version: "encryptionkey@0"
I20260812 02:37:24.068035 8431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "082f2431059a4f998a7068a3abe5b890"
format_stamp: "Formatted at 2026-08-12 02:37:24 on dist-test-slave-rmhg"
server_key: "3536222e6a06ae41a3ce0994a0049477"
server_key_iv: "103710d6cde736fd3de19125fa33b3d1"
server_key_version: "encryptionkey@0"
I20260812 02:37:24.072374 8431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:37:24.075245 8455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:24.076781 8431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 02:37:24.076915 8431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "082f2431059a4f998a7068a3abe5b890"
format_stamp: "Formatted at 2026-08-12 02:37:24 on dist-test-slave-rmhg"
server_key: "3536222e6a06ae41a3ce0994a0049477"
server_key_iv: "103710d6cde736fd3de19125fa33b3d1"
server_key_version: "encryptionkey@0"
I20260812 02:37:24.077044 8431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:24.091679 8431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:24.095368 8431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:24.095582 8431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:24.103231 8431 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:40337
I20260812 02:37:24.103634 8561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:40337 every 8 connection(s)
I20260812 02:37:24.104301 8431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:37:24.107308 8563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:24.108646 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 8431
I20260812 02:37:24.108771 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:37:24.109059 3582 external_mini_cluster.cc:1428] Setting key 1f1c0804402c846b89e423be8a2ebe5d
I20260812 02:37:24.113958 8563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890: Bootstrap starting.
I20260812 02:37:24.116788 8563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:24.117789 8563 log.cc:824] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:37:24 dist-test-slave-rmhg krb5kdc[8414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:24.120128 8563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890: No bootstrap required, opened a new log
I20260812 02:37:24.123972 8563 raft_consensus.cc:348] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082f2431059a4f998a7068a3abe5b890" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40337 } }
I20260812 02:37:24.124151 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:24.110506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36982 (local address 127.3.127.190:40337)
0812 02:37:24.110998 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:24.111021 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:24.111062 (+ 41us) server_negotiation.cc:407] Connection header received
0812 02:37:24.111729 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:24.111748 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:24.112116 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:24.112415 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:24.113455 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:24.114650 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:24.115855 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:24.116371 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:24.116499 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:24.119501 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:24.119506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:24.119523 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:24.119552 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:24.121296 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:24.121752 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:24.121758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:24.121763 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:24.121882 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:24.122150 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:24.122153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:24.122155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:24.122578 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:24.122781 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:24.123063 (+ 282us) server_negotiation.cc:299] Negotiation successful
0812 02:37:24.123369 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":116,"threads_started":1}
I20260812 02:37:24.124224 8563 raft_consensus.cc:374] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:24.124280 8563 raft_consensus.cc:729] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 082f2431059a4f998a7068a3abe5b890, State: Initialized, Role: FOLLOWER
I20260812 02:37:24.124810 8563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082f2431059a4f998a7068a3abe5b890" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40337 } }
I20260812 02:37:24.124985 8563 raft_consensus.cc:388] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:37:24.125052 8563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:37:24.125166 8563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:24.126191 8563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082f2431059a4f998a7068a3abe5b890" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40337 } }
I20260812 02:37:24.126538 8563 leader_election.cc:302] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 082f2431059a4f998a7068a3abe5b890; no voters:
I20260812 02:37:24.126935 8563 leader_election.cc:288] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:37:24.127051 8570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:24.127223 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:37:24.127261 8570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890 [term 1 LEADER]: Becoming Leader. State: Replica: 082f2431059a4f998a7068a3abe5b890, State: Running, Role: LEADER
I20260812 02:37:24.127564 8570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082f2431059a4f998a7068a3abe5b890" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40337 } }
I20260812 02:37:24.128341 8563 sys_catalog.cc:564] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:37:24.128808 8570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 082f2431059a4f998a7068a3abe5b890. Latest consensus state: current_term: 1 leader_uuid: "082f2431059a4f998a7068a3abe5b890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082f2431059a4f998a7068a3abe5b890" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40337 } } }
I20260812 02:37:24.128916 8570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:24.129244 8574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "082f2431059a4f998a7068a3abe5b890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082f2431059a4f998a7068a3abe5b890" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40337 } } }
I20260812 02:37:24.129338 8574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:24.130828 8577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:37:24.133903 8577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:37:24.139892 8577 catalog_manager.cc:1347] Generated new cluster ID: e0477438ceea4e93b963f228b92cf20e
I20260812 02:37:24.139981 8577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:37:24.147962 8577 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:37:24.149184 8577 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:37:24.150789 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:24.144793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36996 (local address 127.3.127.190:40337)
0812 02:37:24.144917 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:24.144921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:24.145237 (+ 316us) server_negotiation.cc:407] Connection header received
0812 02:37:24.145388 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:24.145391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:24.145452 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:24.145546 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:24.146141 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:24.146932 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:24.148040 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:24.148419 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:24.148524 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:24.149642 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:24.149646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:24.149649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:37:24.150269 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:24.150309 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:37:24.150330 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:24.150424 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:24.150548 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 02:37:24.150636 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":46}
I20260812 02:37:24.162160 8577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 082f2431059a4f998a7068a3abe5b890: Generated new TSK 0
I20260812 02:37:24.163002 8577 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:37:24.169349 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 8431
2026-08-12T02:37:24Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1590 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:37:24 dist-test-slave-rmhg krb5kdc[8621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:37:24 dist-test-slave-rmhg krb5kdc[8621](info): set up 2 sockets
Aug 12 02:37:24 dist-test-slave-rmhg krb5kdc[8621](info): commencing operation
krb5kdc: starting...
W20260812 02:37:25.347492 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:37:25 dist-test-slave-rmhg krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:37:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:37:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:37:25.480778 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:38347
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:38347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:25.600116 8645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:25.600428 8645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:25.600499 8645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:25.604655 8645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:25.604735 8645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:25.604763 8645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:25.604784 8645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:25.604804 8645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:37:25.610320 8645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:38347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:38347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:25.611505 8645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:25.612591 8645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:25.621452 8654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:25.621531 8651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:25.621515 8645 server_base.cc:1034] running on GCE node
W20260812 02:37:25.621459 8652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:25.622289 8645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:25.623438 8645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:25.624624 8645 hybrid_clock.cc:648] HybridClock initialized: now 1786502245624605 us; error 31 us; skew 500 ppm
Aug 12 02:37:25 dist-test-slave-rmhg krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502245, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:25.628294 8645 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:37:25.629273 8645 webserver.cc:460] Webserver started at http://127.3.127.190:45923/ using document root <none> and password file <none>
I20260812 02:37:25.629854 8645 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:25.629925 8645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:25.630137 8645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:25.632357 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "11127ff5676f45de919e2aa4ba391604"
format_stamp: "Formatted at 2026-08-12 02:37:25 on dist-test-slave-rmhg"
server_key: "c1a4eb21c6b11473f085a901795ac906"
server_key_iv: "ec84152142a27edda947a2f348135e51"
server_key_version: "encryptionkey@0"
I20260812 02:37:25.632954 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "11127ff5676f45de919e2aa4ba391604"
format_stamp: "Formatted at 2026-08-12 02:37:25 on dist-test-slave-rmhg"
server_key: "c1a4eb21c6b11473f085a901795ac906"
server_key_iv: "ec84152142a27edda947a2f348135e51"
server_key_version: "encryptionkey@0"
I20260812 02:37:25.637017 8645 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260812 02:37:25.639947 8665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:25.641338 8645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:25.641466 8645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "11127ff5676f45de919e2aa4ba391604"
format_stamp: "Formatted at 2026-08-12 02:37:25 on dist-test-slave-rmhg"
server_key: "c1a4eb21c6b11473f085a901795ac906"
server_key_iv: "ec84152142a27edda947a2f348135e51"
server_key_version: "encryptionkey@0"
I20260812 02:37:25.641577 8645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:25.656868 8645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:25.660230 8645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:25.660435 8645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:25.667622 8645 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:38347
I20260812 02:37:25.667886 8761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:38347 every 8 connection(s)
I20260812 02:37:25.668682 8645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:37:25.668923 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 8645
I20260812 02:37:25.669044 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:37:25.669404 3582 external_mini_cluster.cc:1428] Setting key eb8ec10bec9b3e59daaf832b5370e32c
I20260812 02:37:25.672101 8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 02:37:25 dist-test-slave-rmhg krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:25.679061 8762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604: Bootstrap starting.
I20260812 02:37:25.681917 8762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:25.682852 8762 log.cc:824] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:25.684310 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:25.670832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54892 (local address 127.3.127.190:38347)
0812 02:37:25.671277 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:25.671293 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:25.671331 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:37:25.672094 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:25.672125 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:25.672513 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:25.672816 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:25.673768 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:25.674806 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:25.675993 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:25.676497 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:25.676626 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:25.679420 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:25.679427 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:25.679444 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:25.679479 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:25.681345 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:25.681756 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:25.681763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:25.681769 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:25.681913 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:25.682214 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:25.682218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:25.682220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:25.682742 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:25.682944 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:25.683216 (+ 272us) server_negotiation.cc:299] Negotiation successful
0812 02:37:25.683554 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":131,"threads_started":1}
I20260812 02:37:25.684965 8762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604: No bootstrap required, opened a new log
I20260812 02:37:25.688915 8762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11127ff5676f45de919e2aa4ba391604" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38347 } }
I20260812 02:37:25.689199 8762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:25.689266 8762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11127ff5676f45de919e2aa4ba391604, State: Initialized, Role: FOLLOWER
I20260812 02:37:25.689746 8762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11127ff5676f45de919e2aa4ba391604" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38347 } }
I20260812 02:37:25.689918 8762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:37:25.689996 8762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:37:25.690104 8762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:25.691066 8762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11127ff5676f45de919e2aa4ba391604" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38347 } }
I20260812 02:37:25.691393 8762 leader_election.cc:302] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11127ff5676f45de919e2aa4ba391604; no voters:
I20260812 02:37:25.691726 8762 leader_election.cc:288] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:37:25.691870 8768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:25.692102 8768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604 [term 1 LEADER]: Becoming Leader. State: Replica: 11127ff5676f45de919e2aa4ba391604, State: Running, Role: LEADER
I20260812 02:37:25.692402 8768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11127ff5676f45de919e2aa4ba391604" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38347 } }
I20260812 02:37:25.692938 8762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:37:25.694180 8772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11127ff5676f45de919e2aa4ba391604. Latest consensus state: current_term: 1 leader_uuid: "11127ff5676f45de919e2aa4ba391604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11127ff5676f45de919e2aa4ba391604" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38347 } } }
I20260812 02:37:25.694314 8772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:25.694494 8769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11127ff5676f45de919e2aa4ba391604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11127ff5676f45de919e2aa4ba391604" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38347 } } }
I20260812 02:37:25.694657 8769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:25.694888 8783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:37:25.698055 8783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:37:25.703716 8783 catalog_manager.cc:1347] Generated new cluster ID: a0efd7b7884e48acabaf95f320e0b81e
I20260812 02:37:25.703811 8783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:37:25.721052 8783 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:37:25.722131 8783 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:37:25.726099 8783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11127ff5676f45de919e2aa4ba391604: Generated new TSK 0
I20260812 02:37:25.726822 8783 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
I20260812 02:37:25.764312 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38347
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:25.886080 8804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:25.886373 8804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:25.886421 8804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:25.890599 8804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:25.890686 8804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:25.890748 8804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:37:25.896440 8804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:25.897639 8804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:25.898705 8804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:25.908921 8815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:25.908949 8816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:25.909025 8804 server_base.cc:1034] running on GCE node
W20260812 02:37:25.908933 8824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:25.909690 8804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:25.910373 8804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:25.911541 8804 hybrid_clock.cc:648] HybridClock initialized: now 1786502245911519 us; error 34 us; skew 500 ppm
Aug 12 02:37:25 dist-test-slave-rmhg krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502245, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:25.914659 8804 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:37:25.915681 8804 webserver.cc:460] Webserver started at http://127.3.127.129:39423/ using document root <none> and password file <none>
I20260812 02:37:25.916254 8804 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:25.916317 8804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:25.916519 8804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:25.918778 8804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "ae75652990a94757928717317d3d07df"
format_stamp: "Formatted at 2026-08-12 02:37:25 on dist-test-slave-rmhg"
server_key: "6ba781f254b6f5d993088a0939d30b46"
server_key_iv: "21b948799f1beef4c8125107401eeb47"
server_key_version: "encryptionkey@0"
I20260812 02:37:25.919345 8804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "ae75652990a94757928717317d3d07df"
format_stamp: "Formatted at 2026-08-12 02:37:25 on dist-test-slave-rmhg"
server_key: "6ba781f254b6f5d993088a0939d30b46"
server_key_iv: "21b948799f1beef4c8125107401eeb47"
server_key_version: "encryptionkey@0"
I20260812 02:37:25.923347 8804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:37:25.925936 8835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:25.927187 8804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 02:37:25.927312 8804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "ae75652990a94757928717317d3d07df"
format_stamp: "Formatted at 2026-08-12 02:37:25 on dist-test-slave-rmhg"
server_key: "6ba781f254b6f5d993088a0939d30b46"
server_key_iv: "21b948799f1beef4c8125107401eeb47"
server_key_version: "encryptionkey@0"
I20260812 02:37:25.927412 8804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:25.943501 8804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:25.946656 8804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:25.946846 8804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:25.947469 8804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:25.948346 8804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:25.948395 8804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:25.948451 8804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:25.948482 8804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:25.957721 8804 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:32813
I20260812 02:37:25.957723 9018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:32813 every 8 connection(s)
I20260812 02:37:25.958748 8804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:37:25.962565 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 8804
I20260812 02:37:25.962700 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:37:25.962976 3582 external_mini_cluster.cc:1428] Setting key 418dabd87e9cdff3b922a02313f9216c
Aug 12 02:37:25 dist-test-slave-rmhg krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502245, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.127.130@KRBTEST.COM; defaulting to no policy
I20260812 02:37:25.974148 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:25.960696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:49903 (local address 127.3.127.190:38347)
0812 02:37:25.960869 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:25.960874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:25.961790 (+ 916us) server_negotiation.cc:407] Connection header received
0812 02:37:25.963319 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:25.963322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:25.963381 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:25.963465 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:25.964894 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:25.965468 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:25.966836 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:25.967095 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:25.967176 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:25.970211 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:25.970214 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:25.970217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:25.970246 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:25.971662 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:25.972341 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:25.972344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:25.972346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:25.972432 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:25.972822 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:25.972825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:25.972826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:25.973020 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:25.973094 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:25.973841 (+ 747us) server_negotiation.cc:299] Negotiation successful
0812 02:37:25.973993 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 02:37:25.975092 9026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:25.961094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38347 (local address 127.3.127.129:49903)
0812 02:37:25.961613 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:25.961662 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:25.963089 (+ 1427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:25.963666 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:25.963677 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:25.964207 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:25.964749 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:25.964764 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:25.965593 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:25.965596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:25.966691 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:25.966704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:25.967254 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:25.967258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:25.967460 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:25.968115 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:25.968136 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:25.969976 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:25.971796 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:25.971803 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:25.971822 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:25.972218 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:25.972530 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:25.972533 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:25.972535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:25.972731 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:25.973120 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:25.973128 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:25.973558 (+ 430us) client_negotiation.cc:769] Sending connection context
0812 02:37:25.973788 (+ 230us) client_negotiation.cc:240] Negotiation successful
0812 02:37:25.974202 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":123,"threads_started":1}
I20260812 02:37:25.976387 9019 heartbeater.cc:347] Connected to a master server at 127.3.127.190:38347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.130@KRBTEST.COM" created.
I20260812 02:37:25.976684 9019 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:25.977300 9019 heartbeater.cc:511] Master 127.3.127.190:38347 requested a full tablet report, sending...
I20260812 02:37:25.978865 8696 ts_manager.cc:194] Registered new tserver with Master: ae75652990a94757928717317d3d07df (127.3.127.129:32813)
I20260812 02:37:25.980044 8696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:49903
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
I20260812 02:37:25.992642 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38347
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:26.109889 9033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:26.110188 9033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:26.110231 9033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:26.114336 9033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:26.114418 9033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:26.114485 9033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:37:26.120471 9033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:26.121788 9033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:26.122841 9033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:26.132933 9039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:26.133003 9033 server_base.cc:1034] running on GCE node
W20260812 02:37:26.132910 9040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:26.132905 9045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:26.133852 9033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:26.134552 9033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:26.135718 9033 hybrid_clock.cc:648] HybridClock initialized: now 1786502246135695 us; error 37 us; skew 500 ppm
Aug 12 02:37:26 dist-test-slave-rmhg krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502246, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:26.138639 9033 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:37:26.139649 9033 webserver.cc:460] Webserver started at http://127.3.127.130:43055/ using document root <none> and password file <none>
I20260812 02:37:26.140234 9033 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:26.140304 9033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:26.140514 9033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:26.142688 9033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "b9ea479e478d47ddaa740e2cb7940454"
format_stamp: "Formatted at 2026-08-12 02:37:26 on dist-test-slave-rmhg"
server_key: "0b4fbcfad05a375f004cb008a52452da"
server_key_iv: "97518520298f8b9e33da8026003938dc"
server_key_version: "encryptionkey@0"
I20260812 02:37:26.143251 9033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "b9ea479e478d47ddaa740e2cb7940454"
format_stamp: "Formatted at 2026-08-12 02:37:26 on dist-test-slave-rmhg"
server_key: "0b4fbcfad05a375f004cb008a52452da"
server_key_iv: "97518520298f8b9e33da8026003938dc"
server_key_version: "encryptionkey@0"
I20260812 02:37:26.147177 9033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:37:26.149808 9053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:26.151062 9033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 02:37:26.151196 9033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "b9ea479e478d47ddaa740e2cb7940454"
format_stamp: "Formatted at 2026-08-12 02:37:26 on dist-test-slave-rmhg"
server_key: "0b4fbcfad05a375f004cb008a52452da"
server_key_iv: "97518520298f8b9e33da8026003938dc"
server_key_version: "encryptionkey@0"
I20260812 02:37:26.151302 9033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:26.163123 9033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:26.166172 9033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:26.166371 9033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:26.167029 9033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:26.167971 9033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:26.168025 9033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:26.168084 9033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:26.168113 9033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:26.177539 9033 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:34265
I20260812 02:37:26.177568 9244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:34265 every 8 connection(s)
I20260812 02:37:26.178547 9033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:37:26.180351 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 9033
I20260812 02:37:26.180469 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:37:26.180799 3582 external_mini_cluster.cc:1428] Setting key 216596d0fa701d752a669a228f0e78f0
Aug 12 02:37:26 dist-test-slave-rmhg krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502246, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.127.131@KRBTEST.COM; defaulting to no policy
I20260812 02:37:26.195696 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:26.180664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:56781 (local address 127.3.127.190:38347)
0812 02:37:26.180844 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:26.180849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:26.181957 (+ 1108us) server_negotiation.cc:407] Connection header received
0812 02:37:26.183509 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:26.183512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:26.183570 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:26.183656 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:26.185442 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:26.186237 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:26.188027 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:26.188253 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:26.188333 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:26.191575 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:26.191580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:26.191583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:26.191621 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:26.193268 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:26.193865 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:26.193869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:26.193871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:26.193980 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:26.194417 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:26.194421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:26.194423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:26.194636 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:26.194731 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:26.195427 (+ 696us) server_negotiation.cc:299] Negotiation successful
0812 02:37:26.195562 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.131@KRBTEST.COM" created.
I20260812 02:37:26.196933 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:26.181120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38347 (local address 127.3.127.130:56781)
0812 02:37:26.181777 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:26.181843 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:26.183273 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:26.183874 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:26.183883 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:26.184378 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:26.185242 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:26.185270 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:26.186412 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:26.186416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:26.187859 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:26.187874 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:26.188444 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:26.188449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:26.188697 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:26.189417 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:26.189442 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:26.191348 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:26.193403 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:26.193409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:26.193426 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:26.193740 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:26.194106 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:26.194110 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:26.194112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:26.194307 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:26.194744 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:26.194753 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:26.195097 (+ 344us) client_negotiation.cc:769] Sending connection context
0812 02:37:26.195325 (+ 228us) client_negotiation.cc:240] Negotiation successful
0812 02:37:26.195682 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":157,"threads_started":1}
I20260812 02:37:26.198328 9245 heartbeater.cc:347] Connected to a master server at 127.3.127.190:38347
I20260812 02:37:26.198657 9245 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:26.199355 9245 heartbeater.cc:511] Master 127.3.127.190:38347 requested a full tablet report, sending...
I20260812 02:37:26.200721 8696 ts_manager.cc:194] Registered new tserver with Master: b9ea479e478d47ddaa740e2cb7940454 (127.3.127.130:34265)
I20260812 02:37:26.201407 8696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:56781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
I20260812 02:37:26.215890 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38347
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:26.336256 9253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:26.336570 9253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:26.336627 9253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:26.340969 9253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:26.341056 9253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:26.341125 9253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:37:26.346973 9253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:26.348157 9253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:26.349279 9253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:26.359319 9268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:26.359357 9262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:26.359321 9264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:26.359730 9253 server_base.cc:1034] running on GCE node
I20260812 02:37:26.360188 9253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:26.360874 9253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:26.362058 9253 hybrid_clock.cc:648] HybridClock initialized: now 1786502246362020 us; error 61 us; skew 500 ppm
Aug 12 02:37:26 dist-test-slave-rmhg krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502246, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:26.364961 9253 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:37:26.365954 9253 webserver.cc:460] Webserver started at http://127.3.127.131:45951/ using document root <none> and password file <none>
I20260812 02:37:26.366531 9253 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:26.366592 9253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:26.366770 9253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:26.368904 9253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "2f4d20d715dd4625bf35bfcc8da2cd07"
format_stamp: "Formatted at 2026-08-12 02:37:26 on dist-test-slave-rmhg"
server_key: "1c00b605dbe21ee988d4ee856bcc7714"
server_key_iv: "bea5a923c638d11f182a83c8feacb0c7"
server_key_version: "encryptionkey@0"
I20260812 02:37:26.369505 9253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "2f4d20d715dd4625bf35bfcc8da2cd07"
format_stamp: "Formatted at 2026-08-12 02:37:26 on dist-test-slave-rmhg"
server_key: "1c00b605dbe21ee988d4ee856bcc7714"
server_key_iv: "bea5a923c638d11f182a83c8feacb0c7"
server_key_version: "encryptionkey@0"
I20260812 02:37:26.373600 9253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:37:26.376490 9276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:26.378230 9253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:26.378379 9253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "2f4d20d715dd4625bf35bfcc8da2cd07"
format_stamp: "Formatted at 2026-08-12 02:37:26 on dist-test-slave-rmhg"
server_key: "1c00b605dbe21ee988d4ee856bcc7714"
server_key_iv: "bea5a923c638d11f182a83c8feacb0c7"
server_key_version: "encryptionkey@0"
I20260812 02:37:26.378516 9253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:26.394829 9253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:26.397814 9253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:26.397979 9253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:26.398593 9253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:26.399466 9253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:26.399513 9253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:26.399554 9253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:26.399575 9253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:26.408507 9253 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:43173
I20260812 02:37:26.408540 9459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:43173 every 8 connection(s)
I20260812 02:37:26.409612 9253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:37:26.414307 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 9253
I20260812 02:37:26.414444 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:37:26.414770 3582 external_mini_cluster.cc:1428] Setting key 362a9c2ff1c834c3a2fec4af41e65d3e
Aug 12 02:37:26 dist-test-slave-rmhg krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502246, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:26.423694 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:26.411570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:39791 (local address 127.3.127.190:38347)
0812 02:37:26.411721 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:26.411724 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:26.412638 (+ 914us) server_negotiation.cc:407] Connection header received
0812 02:37:26.414071 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:26.414074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:26.414151 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:26.414241 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:26.415865 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:26.416484 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:26.417824 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:26.418047 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:26.418127 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:26.420597 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:26.420601 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:26.420604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:26.420629 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:26.421987 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:26.422429 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:26.422431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:26.422432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:26.422502 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:26.422756 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:26.422758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:26.422759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:26.422931 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:26.423001 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:26.423462 (+ 461us) server_negotiation.cc:299] Negotiation successful
0812 02:37:26.423567 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 02:37:26.424602 9465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:26.412009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38347 (local address 127.3.127.131:39791)
0812 02:37:26.412473 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:26.412519 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:26.413845 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:26.414479 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:26.414489 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:26.414982 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:26.415576 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:26.415597 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:26.416615 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:26.416618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:26.417702 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:26.417712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:26.418185 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:26.418188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:26.418342 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:26.418948 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:26.418965 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:26.420433 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:26.422073 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:26.422077 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:26.422091 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:26.422356 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:26.422561 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:26.422563 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:26.422565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:26.422694 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:26.423001 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:26.423005 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:26.423270 (+ 265us) client_negotiation.cc:769] Sending connection context
0812 02:37:26.423430 (+ 160us) client_negotiation.cc:240] Negotiation successful
0812 02:37:26.423707 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":121,"threads_started":1}
I20260812 02:37:26.425912 9460 heartbeater.cc:347] Connected to a master server at 127.3.127.190:38347
I20260812 02:37:26.426154 9460 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:26.426656 9460 heartbeater.cc:511] Master 127.3.127.190:38347 requested a full tablet report, sending...
I20260812 02:37:26.427771 8696 ts_manager.cc:194] Registered new tserver with Master: 2f4d20d715dd4625bf35bfcc8da2cd07 (127.3.127.131:43173)
I20260812 02:37:26.428363 8696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:39791
I20260812 02:37:26.428891 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 02:37:26 dist-test-slave-rmhg krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502246, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 02:37:26 dist-test-slave-rmhg krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502246, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:26.459937 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:26.451054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54902 (local address 127.3.127.190:38347)
0812 02:37:26.451224 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:26.451229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:26.451384 (+ 155us) server_negotiation.cc:407] Connection header received
0812 02:37:26.451591 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:26.451594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:26.451641 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:26.451710 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:26.452477 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:26.453020 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:26.453960 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:26.454187 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:26.454283 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:26.456707 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:26.456710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:26.456712 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:26.456744 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:26.458169 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:26.458635 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:26.458639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:26.458640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:26.458726 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:26.458988 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:26.458992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:26.458993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:26.459249 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:26.459377 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:26.459623 (+ 246us) server_negotiation.cc:299] Negotiation successful
0812 02:37:26.459751 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260812 02:37:26.460367 8692 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54902
I20260812 02:37:26.461341 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 8804
I20260812 02:37:26.480319 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 9033
I20260812 02:37:26.496299 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 9253
I20260812 02:37:26.513463 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 8645
2026-08-12T02:37:26Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2341 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:37:26 dist-test-slave-rmhg krb5kdc[9480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:37:26 dist-test-slave-rmhg krb5kdc[9480](info): set up 2 sockets
Aug 12 02:37:26 dist-test-slave-rmhg krb5kdc[9480](info): commencing operation
krb5kdc: starting...
W20260812 02:37:27.688674 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:37:27 dist-test-slave-rmhg krb5kdc[9480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:37:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:37:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:37:27.819985 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:33591
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:39427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:33591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:27.935112 9504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:27.935418 9504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:27.935478 9504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:27.939574 9504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:27.939649 9504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:27.939675 9504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:27.939698 9504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:27.939718 9504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:37:27.945313 9504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:33591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:33591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:27.946530 9504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:27.947618 9504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:27.956992 9512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:27.956962 9513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:27.956975 9515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:27.957571 9504 server_base.cc:1034] running on GCE node
I20260812 02:37:27.958214 9504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:27.959439 9504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:27.960608 9504 hybrid_clock.cc:648] HybridClock initialized: now 1786502247960591 us; error 31 us; skew 500 ppm
Aug 12 02:37:27 dist-test-slave-rmhg krb5kdc[9480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502247, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:27.963471 9504 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:37:27.964479 9504 webserver.cc:460] Webserver started at http://127.3.127.190:41395/ using document root <none> and password file <none>
I20260812 02:37:27.965046 9504 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:27.965116 9504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:27.965373 9504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:27.967505 9504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "56562bf87ffc4489bef0341d15c49845"
format_stamp: "Formatted at 2026-08-12 02:37:27 on dist-test-slave-rmhg"
server_key: "e8dc852ffe6664081fdca4cce5deda01"
server_key_iv: "57f6d56923ef3c95cfe5227530205750"
server_key_version: "encryptionkey@0"
I20260812 02:37:27.968063 9504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "56562bf87ffc4489bef0341d15c49845"
format_stamp: "Formatted at 2026-08-12 02:37:27 on dist-test-slave-rmhg"
server_key: "e8dc852ffe6664081fdca4cce5deda01"
server_key_iv: "57f6d56923ef3c95cfe5227530205750"
server_key_version: "encryptionkey@0"
I20260812 02:37:27.971891 9504 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 02:37:27.974364 9529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:27.975687 9504 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 02:37:27.975824 9504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "56562bf87ffc4489bef0341d15c49845"
format_stamp: "Formatted at 2026-08-12 02:37:27 on dist-test-slave-rmhg"
server_key: "e8dc852ffe6664081fdca4cce5deda01"
server_key_iv: "57f6d56923ef3c95cfe5227530205750"
server_key_version: "encryptionkey@0"
I20260812 02:37:27.975927 9504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:27.985898 9504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:27.989039 9504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:27.989269 9504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:27.996429 9504 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:33591
I20260812 02:37:27.996449 9611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:33591 every 8 connection(s)
I20260812 02:37:27.997503 9504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:37:28.000402 9613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:28.006486 9613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845: Bootstrap starting.
I20260812 02:37:28.007838 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 9504
I20260812 02:37:28.007979 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:37:28.008294 3582 external_mini_cluster.cc:1428] Setting key c2f6af05d44c4e2235f68ee6cff4f02b
I20260812 02:37:28.009763 9613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:28.010619 9613 log.cc:824] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:28.013139 9613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845: No bootstrap required, opened a new log
I20260812 02:37:28.017758 9613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56562bf87ffc4489bef0341d15c49845" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33591 } }
I20260812 02:37:28.018018 9613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Aug 12 02:37:28 dist-test-slave-rmhg krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:28.018082 9613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56562bf87ffc4489bef0341d15c49845, State: Initialized, Role: FOLLOWER
I20260812 02:37:28.018656 9613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56562bf87ffc4489bef0341d15c49845" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33591 } }
I20260812 02:37:28.018836 9613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:37:28.018908 9613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:37:28.019012 9613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:28.019956 9613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56562bf87ffc4489bef0341d15c49845" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33591 } }
I20260812 02:37:28.020313 9613 leader_election.cc:302] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56562bf87ffc4489bef0341d15c49845; no voters:
I20260812 02:37:28.020682 9613 leader_election.cc:288] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:37:28.020812 9620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:28.021055 9620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845 [term 1 LEADER]: Becoming Leader. State: Replica: 56562bf87ffc4489bef0341d15c49845, State: Running, Role: LEADER
I20260812 02:37:28.021422 9620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56562bf87ffc4489bef0341d15c49845" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33591 } }
I20260812 02:37:28.022248 9613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:37:28.023037 9621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56562bf87ffc4489bef0341d15c49845" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56562bf87ffc4489bef0341d15c49845" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33591 } } }
I20260812 02:37:28.023183 9621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:28.023478 9622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56562bf87ffc4489bef0341d15c49845. Latest consensus state: current_term: 1 leader_uuid: "56562bf87ffc4489bef0341d15c49845" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56562bf87ffc4489bef0341d15c49845" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33591 } } }
I20260812 02:37:28.023574 9622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:28.024874 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:28.009668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34774 (local address 127.3.127.190:33591)
0812 02:37:28.010149 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:28.010167 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:28.010207 (+ 40us) server_negotiation.cc:407] Connection header received
0812 02:37:28.010952 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:28.010980 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:28.011455 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:28.011785 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:28.012685 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:28.013905 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:28.015117 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:28.015664 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:28.015795 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:28.018726 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:28.018731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:28.018747 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:28.018779 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:28.020561 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:28.021109 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:28.021116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:28.021122 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:28.021309 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:28.021629 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:28.021632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:28.021634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:28.022185 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:28.022398 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:28.023154 (+ 756us) server_negotiation.cc:299] Negotiation successful
0812 02:37:28.023807 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":145,"threads_started":1}
I20260812 02:37:28.025027 9629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:37:28.028635 9629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:37:28.034631 9629 catalog_manager.cc:1347] Generated new cluster ID: 678f9eeedc3b4ddd8da60e123441a5d5
I20260812 02:37:28.034709 9629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:37:28.041985 9629 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:37:28.043337 9629 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:37:28.053699 9629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56562bf87ffc4489bef0341d15c49845: Generated new TSK 0
I20260812 02:37:28.054627 9629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
I20260812 02:37:28.090358 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33591
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:39427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:28.210631 9659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:28.210952 9659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:28.211018 9659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:28.215365 9659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:28.215451 9659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:28.215540 9659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:37:28.221273 9659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:28.222584 9659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:28.223912 9659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:28.234186 9669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:28.234191 9675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:28.234187 9667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:28.234457 9659 server_base.cc:1034] running on GCE node
I20260812 02:37:28.234966 9659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:28.235677 9659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:28.236871 9659 hybrid_clock.cc:648] HybridClock initialized: now 1786502248236838 us; error 42 us; skew 500 ppm
Aug 12 02:37:28 dist-test-slave-rmhg krb5kdc[9480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:28.239812 9659 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:37:28.240841 9659 webserver.cc:460] Webserver started at http://127.3.127.129:36661/ using document root <none> and password file <none>
I20260812 02:37:28.241485 9659 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:28.241560 9659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:28.241780 9659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:28.243920 9659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "d364be222f894e0db980094bf6777129"
format_stamp: "Formatted at 2026-08-12 02:37:28 on dist-test-slave-rmhg"
server_key: "08f3ff2213dfeabbb778fbfa49d173fd"
server_key_iv: "648dc09837dde0ab07b05d2f3e7a1528"
server_key_version: "encryptionkey@0"
I20260812 02:37:28.244532 9659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "d364be222f894e0db980094bf6777129"
format_stamp: "Formatted at 2026-08-12 02:37:28 on dist-test-slave-rmhg"
server_key: "08f3ff2213dfeabbb778fbfa49d173fd"
server_key_iv: "648dc09837dde0ab07b05d2f3e7a1528"
server_key_version: "encryptionkey@0"
I20260812 02:37:28.248697 9659 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260812 02:37:28.251511 9684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:28.252879 9659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:37:28.253017 9659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "d364be222f894e0db980094bf6777129"
format_stamp: "Formatted at 2026-08-12 02:37:28 on dist-test-slave-rmhg"
server_key: "08f3ff2213dfeabbb778fbfa49d173fd"
server_key_iv: "648dc09837dde0ab07b05d2f3e7a1528"
server_key_version: "encryptionkey@0"
I20260812 02:37:28.253145 9659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:28.271067 9659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:28.274307 9659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:28.274505 9659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:28.275139 9659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:28.276023 9659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:28.276079 9659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:28.276137 9659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:28.276161 9659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:28.285137 9659 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:35827
I20260812 02:37:28.285213 9872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:35827 every 8 connection(s)
I20260812 02:37:28.286150 9659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:37:28.288416 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 9659
I20260812 02:37:28.288559 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:37:28.288888 3582 external_mini_cluster.cc:1428] Setting key 22d9d50839f5c0919d52d1d063fb59d7
Aug 12 02:37:28 dist-test-slave-rmhg krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.127.130@KRBTEST.COM; defaulting to no policy
I20260812 02:37:28.301584 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:28.288149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:44989 (local address 127.3.127.190:33591)
0812 02:37:28.288304 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:28.288307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:28.289485 (+ 1178us) server_negotiation.cc:407] Connection header received
0812 02:37:28.291085 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:28.291089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:28.291154 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:28.291228 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:28.292511 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:28.293052 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:28.294534 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:28.294782 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:28.294881 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:28.297639 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:28.297642 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:28.297644 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:28.297672 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:28.299255 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:28.299875 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:28.299878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:28.299880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:28.299966 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:28.300331 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:28.300334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:28.300336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:28.300546 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:28.300637 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:28.301253 (+ 616us) server_negotiation.cc:299] Negotiation successful
0812 02:37:28.301405 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.130@KRBTEST.COM" created.
I20260812 02:37:28.302760 9875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:28.288639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:33591 (local address 127.3.127.129:44989)
0812 02:37:28.289276 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:28.289340 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:28.290869 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:28.291425 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:28.291433 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:28.291830 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:28.292358 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:28.292372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:28.293345 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:28.293349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:28.294350 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:28.294361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:28.294957 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:28.294962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:28.295177 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:28.295806 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:28.295824 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:28.297422 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:28.299396 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:28.299404 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:28.299424 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:28.299759 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:28.300056 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:28.300060 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:28.300062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:28.300236 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:28.300657 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:28.300673 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:28.300966 (+ 293us) client_negotiation.cc:769] Sending connection context
0812 02:37:28.301183 (+ 217us) client_negotiation.cc:240] Negotiation successful
0812 02:37:28.301580 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":148,"threads_started":1}
I20260812 02:37:28.304536 9873 heartbeater.cc:347] Connected to a master server at 127.3.127.190:33591
I20260812 02:37:28.304840 9873 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:28.305521 9873 heartbeater.cc:511] Master 127.3.127.190:33591 requested a full tablet report, sending...
I20260812 02:37:28.307634 9556 ts_manager.cc:194] Registered new tserver with Master: d364be222f894e0db980094bf6777129 (127.3.127.129:35827)
I20260812 02:37:28.309405 9556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:44989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
I20260812 02:37:28.320355 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33591
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:39427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:28.441715 9880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:28.442022 9880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:28.442090 9880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:28.446285 9880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:28.446367 9880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:28.446452 9880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:37:28.452556 9880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:28.453830 9880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:28.454929 9880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:28.465029 9891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:28.465029 9890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:28.465076 9895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:28.465325 9880 server_base.cc:1034] running on GCE node
I20260812 02:37:28.465843 9880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:28.466557 9880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:28.467729 9880 hybrid_clock.cc:648] HybridClock initialized: now 1786502248467698 us; error 48 us; skew 500 ppm
Aug 12 02:37:28 dist-test-slave-rmhg krb5kdc[9480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:28.470731 9880 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:37:28.471686 9880 webserver.cc:460] Webserver started at http://127.3.127.130:41899/ using document root <none> and password file <none>
I20260812 02:37:28.472263 9880 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:28.472333 9880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:28.472602 9880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:28.474937 9880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "16317e3d778649869fc70655c16231ae"
format_stamp: "Formatted at 2026-08-12 02:37:28 on dist-test-slave-rmhg"
server_key: "e6a2f1dd9eea48f27c80b838846068e3"
server_key_iv: "d6813753ed7606dfccb93b25a4980b39"
server_key_version: "encryptionkey@0"
I20260812 02:37:28.475526 9880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "16317e3d778649869fc70655c16231ae"
format_stamp: "Formatted at 2026-08-12 02:37:28 on dist-test-slave-rmhg"
server_key: "e6a2f1dd9eea48f27c80b838846068e3"
server_key_iv: "d6813753ed7606dfccb93b25a4980b39"
server_key_version: "encryptionkey@0"
I20260812 02:37:28.479518 9880 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 02:37:28.482338 9904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:28.483635 9880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:28.483781 9880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "16317e3d778649869fc70655c16231ae"
format_stamp: "Formatted at 2026-08-12 02:37:28 on dist-test-slave-rmhg"
server_key: "e6a2f1dd9eea48f27c80b838846068e3"
server_key_iv: "d6813753ed7606dfccb93b25a4980b39"
server_key_version: "encryptionkey@0"
I20260812 02:37:28.483891 9880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:28.500955 9880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:28.504477 9880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:28.504693 9880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:28.505390 9880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:28.506345 9880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:28.506397 9880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:28.506453 9880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:28.506482 9880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:28.515805 9880 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:35839
I20260812 02:37:28.515874 10093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:35839 every 8 connection(s)
I20260812 02:37:28.516824 9880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:37:28.518900 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 9880
I20260812 02:37:28.519052 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:37:28.519335 3582 external_mini_cluster.cc:1428] Setting key cc88dbf7b4c062d856aa9212ae4a42c9
Aug 12 02:37:28 dist-test-slave-rmhg krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.127.131@KRBTEST.COM; defaulting to no policy
I20260812 02:37:28.532629 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:28.518798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:33377 (local address 127.3.127.190:33591)
0812 02:37:28.518952 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:28.518956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:28.519881 (+ 925us) server_negotiation.cc:407] Connection header received
0812 02:37:28.521496 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:28.521500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:28.521557 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:28.521626 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:28.523049 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:28.523755 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:28.525265 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:28.525599 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:28.525707 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:28.528721 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:28.528726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:28.528729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:28.528760 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:28.530351 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:28.530913 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:28.530917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:28.530919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:28.531042 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:28.531485 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:28.531488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:28.531489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:28.531722 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:28.531816 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:28.532377 (+ 561us) server_negotiation.cc:299] Negotiation successful
0812 02:37:28.532494 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.131@KRBTEST.COM" created.
I20260812 02:37:28.533937 10099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:28.519211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:33591 (local address 127.3.127.130:33377)
0812 02:37:28.519719 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:28.519776 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:28.521201 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:28.521829 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:28.521837 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:28.522238 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:28.522875 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:28.522900 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:28.523892 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:28.523895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:28.525097 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:28.525109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:28.525813 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:28.525818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:28.525996 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:28.526780 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:28.526808 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:28.528514 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:28.530476 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:28.530482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:28.530497 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:28.530775 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:28.531188 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:28.531191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:28.531193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:28.531361 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:28.531854 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:28.531860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:28.532141 (+ 281us) client_negotiation.cc:769] Sending connection context
0812 02:37:28.532340 (+ 199us) client_negotiation.cc:240] Negotiation successful
0812 02:37:28.532638 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":132,"threads_started":1}
I20260812 02:37:28.536823 10096 heartbeater.cc:347] Connected to a master server at 127.3.127.190:33591
I20260812 02:37:28.537106 10096 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:28.537710 10096 heartbeater.cc:511] Master 127.3.127.190:33591 requested a full tablet report, sending...
I20260812 02:37:28.538882 9556 ts_manager.cc:194] Registered new tserver with Master: 16317e3d778649869fc70655c16231ae (127.3.127.130:35839)
I20260812 02:37:28.539507 9556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:33377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
I20260812 02:37:28.550251 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33591
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:39427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:28.669989 10106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:28.670279 10106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:28.670329 10106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:28.674517 10106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:28.674638 10106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:28.674715 10106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:37:28.680387 10106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:28.681576 10106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:28.682639 10106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:28.692690 10115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:28.692696 10116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:28.692730 10121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:28.693074 10106 server_base.cc:1034] running on GCE node
I20260812 02:37:28.693532 10106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:28.694259 10106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:28.695429 10106 hybrid_clock.cc:648] HybridClock initialized: now 1786502248695391 us; error 54 us; skew 500 ppm
Aug 12 02:37:28 dist-test-slave-rmhg krb5kdc[9480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:28.698421 10106 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:37:28.699460 10106 webserver.cc:460] Webserver started at http://127.3.127.131:43375/ using document root <none> and password file <none>
I20260812 02:37:28.700062 10106 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:28.700136 10106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:28.700352 10106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:28.702688 10106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "3eea281d6a8c4960919ec436aa33a494"
format_stamp: "Formatted at 2026-08-12 02:37:28 on dist-test-slave-rmhg"
server_key: "2309f21f1216bd6f4e8d2defd52db8cd"
server_key_iv: "b095040aa438152a4475736b3c77b50b"
server_key_version: "encryptionkey@0"
I20260812 02:37:28.703341 10106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "3eea281d6a8c4960919ec436aa33a494"
format_stamp: "Formatted at 2026-08-12 02:37:28 on dist-test-slave-rmhg"
server_key: "2309f21f1216bd6f4e8d2defd52db8cd"
server_key_iv: "b095040aa438152a4475736b3c77b50b"
server_key_version: "encryptionkey@0"
I20260812 02:37:28.707552 10106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:37:28.710481 10135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:28.711886 10106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:28.712025 10106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "3eea281d6a8c4960919ec436aa33a494"
format_stamp: "Formatted at 2026-08-12 02:37:28 on dist-test-slave-rmhg"
server_key: "2309f21f1216bd6f4e8d2defd52db8cd"
server_key_iv: "b095040aa438152a4475736b3c77b50b"
server_key_version: "encryptionkey@0"
I20260812 02:37:28.712152 10106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:28.730471 10106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:28.734059 10106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:28.734269 10106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:28.734921 10106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:28.735838 10106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:28.735890 10106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:28.735945 10106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:28.735971 10106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:28.745347 10106 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:38707
I20260812 02:37:28.745379 10299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:38707 every 8 connection(s)
I20260812 02:37:28.746366 10106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:37:28.748293 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 10106
I20260812 02:37:28.748425 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:37:28.748746 3582 external_mini_cluster.cc:1428] Setting key 0923d835383c974564a707c5ff0792e7
Aug 12 02:37:28 dist-test-slave-rmhg krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:28.762625 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:28.748488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:46167 (local address 127.3.127.190:33591)
0812 02:37:28.748659 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:28.748664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:28.749812 (+ 1148us) server_negotiation.cc:407] Connection header received
0812 02:37:28.751540 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:28.751543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:28.751592 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:28.751659 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:28.752933 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:28.753606 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:28.754854 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:28.755128 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:28.755228 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:28.758212 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:28.758215 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:28.758217 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:28.758249 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:28.759819 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:28.760626 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:28.760630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:28.760631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:28.760738 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:28.761212 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:28.761217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:28.761218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:28.761457 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:28.761611 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:28.762295 (+ 684us) server_negotiation.cc:299] Negotiation successful
0812 02:37:28.762457 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 02:37:28.763640 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:28.749002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:33591 (local address 127.3.127.131:46167)
0812 02:37:28.749643 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:28.749714 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:28.751327 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:28.751811 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:28.751819 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:28.752216 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:28.752757 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:28.752780 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:28.753747 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:28.753750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:28.754711 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:28.754722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:28.755319 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:28.755322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:28.755502 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:28.756203 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:28.756223 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:28.757985 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:28.759983 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:28.759991 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:28.760008 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:28.760359 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:28.760847 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:28.760850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:28.760851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:28.761036 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:28.761649 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:28.761656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:28.761970 (+ 314us) client_negotiation.cc:769] Sending connection context
0812 02:37:28.762204 (+ 234us) client_negotiation.cc:240] Negotiation successful
0812 02:37:28.762575 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":198,"threads_started":1}
I20260812 02:37:28.765027 10300 heartbeater.cc:347] Connected to a master server at 127.3.127.190:33591
I20260812 02:37:28.765374 10300 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:28.765957 10300 heartbeater.cc:511] Master 127.3.127.190:33591 requested a full tablet report, sending...
I20260812 02:37:28.767189 9556 ts_manager.cc:194] Registered new tserver with Master: 3eea281d6a8c4960919ec436aa33a494 (127.3.127.131:38707)
I20260812 02:37:28.767853 9556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:46167
I20260812 02:37:28.775965 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 02:37:28 dist-test-slave-rmhg krb5kdc[9480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 02:37:28 dist-test-slave-rmhg krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.129@KRBTEST.COM
I20260812 02:37:28.806265 10312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:28.795820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58088 (local address 127.3.127.129:35827)
0812 02:37:28.796202 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:28.796220 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:28.796252 (+ 32us) server_negotiation.cc:407] Connection header received
0812 02:37:28.796315 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:28.796321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:28.796526 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:28.796666 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:28.797524 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:28.798361 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:28.799260 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:28.799681 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:28.799791 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:28.802484 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:28.802504 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:28.802523 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:28.802560 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:28.804337 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:28.804865 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:28.804870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:28.804872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:28.804963 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:28.805212 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:28.805216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:28.805217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:28.805509 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:28.805699 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:28.805951 (+ 252us) server_negotiation.cc:299] Negotiation successful
0812 02:37:28.806107 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":101,"threads_started":1}
Aug 12 02:37:28 dist-test-slave-rmhg krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:28.819085 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:28.809876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34788 (local address 127.3.127.190:33591)
0812 02:37:28.810107 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:28.810111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:28.810126 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:37:28.810176 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:28.810179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:28.810225 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:28.810302 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:28.811018 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:28.811593 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:28.812476 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:28.812766 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:28.812865 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:28.815674 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:28.815678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:28.815680 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:28.815717 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:28.817223 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:28.817708 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:28.817712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:28.817713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:28.817808 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:28.818103 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:28.818107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:28.818109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:28.818371 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:28.818505 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:28.818769 (+ 264us) server_negotiation.cc:299] Negotiation successful
0812 02:37:28.818917 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260812 02:37:28.819519 9556 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34788
I20260812 02:37:28.820276 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 9659
I20260812 02:37:28.835979 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 9880
I20260812 02:37:28.852353 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 10106
I20260812 02:37:28.870903 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 9504
2026-08-12T02:37:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2360 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:37:28 dist-test-slave-rmhg krb5kdc[10321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:37:28 dist-test-slave-rmhg krb5kdc[10321](info): set up 2 sockets
Aug 12 02:37:28 dist-test-slave-rmhg krb5kdc[10321](info): commencing operation
krb5kdc: starting...
W20260812 02:37:30.077876 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.170s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:37:30 dist-test-slave-rmhg krb5kdc[10321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:37:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:37:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:37:30.224502 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:36009
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:41871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:36009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:30.345781 10344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:30.346088 10344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:30.346131 10344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:30.350204 10344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:30.350296 10344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:30.350315 10344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:30.350332 10344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:30.350345 10344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:37:30.356038 10344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:41871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:36009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:36009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:30.357264 10344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:30.358321 10344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:30.367194 10353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:30.367197 10351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:30.367197 10349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:30.367643 10344 server_base.cc:1034] running on GCE node
I20260812 02:37:30.368279 10344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:30.369602 10344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:30.370780 10344 hybrid_clock.cc:648] HybridClock initialized: now 1786502250370761 us; error 32 us; skew 500 ppm
Aug 12 02:37:30 dist-test-slave-rmhg krb5kdc[10321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502250, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:30.373845 10344 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:37:30.374653 10344 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:30.374728 10344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:30.374951 10344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:30.377722 10344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "c5f8c0e7a827457a953245163fde5735"
format_stamp: "Formatted at 2026-08-12 02:37:30 on dist-test-slave-rmhg"
server_key: "4d26aaf446bdacde81cb46e12bea0fe6"
server_key_iv: "cdf0b75698ceaccbc0a6db387f953c2c"
server_key_version: "encryptionkey@0"
I20260812 02:37:30.378494 10344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "c5f8c0e7a827457a953245163fde5735"
format_stamp: "Formatted at 2026-08-12 02:37:30 on dist-test-slave-rmhg"
server_key: "4d26aaf446bdacde81cb46e12bea0fe6"
server_key_iv: "cdf0b75698ceaccbc0a6db387f953c2c"
server_key_version: "encryptionkey@0"
I20260812 02:37:30.383136 10344 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:37:30.385998 10360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:30.387362 10344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:37:30.387506 10344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "c5f8c0e7a827457a953245163fde5735"
format_stamp: "Formatted at 2026-08-12 02:37:30 on dist-test-slave-rmhg"
server_key: "4d26aaf446bdacde81cb46e12bea0fe6"
server_key_iv: "cdf0b75698ceaccbc0a6db387f953c2c"
server_key_version: "encryptionkey@0"
I20260812 02:37:30.387642 10344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:30.401976 10344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:30.405392 10344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:30.405591 10344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:30.412891 10344 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:36009
I20260812 02:37:30.413187 10451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:36009 every 8 connection(s)
I20260812 02:37:30.413681 10344 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:37:30.416807 10452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:30.423110 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 10344
I20260812 02:37:30.423111 10452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735: Bootstrap starting.
I20260812 02:37:30.423255 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:37:30.423513 3582 external_mini_cluster.cc:1428] Setting key 670c80de6c9786f4abe16ccb01c025cc
I20260812 02:37:30.426360 10452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:30.427423 10452 log.cc:824] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:30.430123 10452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735: No bootstrap required, opened a new log
Aug 12 02:37:30 dist-test-slave-rmhg krb5kdc[10321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:30.434623 10452 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f8c0e7a827457a953245163fde5735" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36009 } }
I20260812 02:37:30.434969 10452 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:30.435041 10452 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5f8c0e7a827457a953245163fde5735, State: Initialized, Role: FOLLOWER
I20260812 02:37:30.435518 10452 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f8c0e7a827457a953245163fde5735" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36009 } }
I20260812 02:37:30.435664 10452 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:37:30.435735 10452 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:37:30.435837 10452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:30.436769 10452 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f8c0e7a827457a953245163fde5735" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36009 } }
I20260812 02:37:30.437129 10452 leader_election.cc:302] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5f8c0e7a827457a953245163fde5735; no voters:
I20260812 02:37:30.437587 10452 leader_election.cc:288] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:37:30.437665 10458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:30.437891 10458 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735 [term 1 LEADER]: Becoming Leader. State: Replica: c5f8c0e7a827457a953245163fde5735, State: Running, Role: LEADER
I20260812 02:37:30.438223 10458 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f8c0e7a827457a953245163fde5735" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36009 } }
I20260812 02:37:30.439038 10452 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:37:30.440064 10460 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5f8c0e7a827457a953245163fde5735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f8c0e7a827457a953245163fde5735" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36009 } } }
I20260812 02:37:30.440202 10460 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:30.440502 10459 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5f8c0e7a827457a953245163fde5735. Latest consensus state: current_term: 1 leader_uuid: "c5f8c0e7a827457a953245163fde5735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f8c0e7a827457a953245163fde5735" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36009 } } }
I20260812 02:37:30.440589 10459 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:30.440734 10474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:37:30.440757 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:30.425313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35078 (local address 127.3.127.190:36009)
0812 02:37:30.425830 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:30.425851 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:30.425897 (+ 46us) server_negotiation.cc:407] Connection header received
0812 02:37:30.426856 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:30.426884 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:30.427390 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:30.427804 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:30.428734 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:30.430239 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:30.431460 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:30.431977 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:30.432138 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:30.435065 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:30.435092 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:30.435113 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:30.435150 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:30.437007 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:30.437530 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:30.437536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:30.437541 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:30.437708 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:30.438080 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:30.438084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:30.438086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:30.438651 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:30.438880 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:30.439270 (+ 390us) server_negotiation.cc:299] Negotiation successful
0812 02:37:30.439632 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":135,"threads_started":1}
I20260812 02:37:30.444339 10474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:37:30.450650 10474 catalog_manager.cc:1347] Generated new cluster ID: 4b8096a8293a45b5aa510bcca853b695
I20260812 02:37:30.450732 10474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:37:30.493455 10474 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:37:30.494585 10474 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:37:30.501765 10474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735: Generated new TSK 0
I20260812 02:37:30.502604 10474 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
I20260812 02:37:30.546480 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:36009
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:41871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:30.668568 10492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:30.668885 10492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:30.668934 10492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:30.673226 10492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:30.673311 10492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:30.673375 10492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:37:30.678963 10492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:41871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:36009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:30.680099 10492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:30.681111 10492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:30.690665 10503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:30.690665 10501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:30.690680 10508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:30.690893 10492 server_base.cc:1034] running on GCE node
I20260812 02:37:30.691599 10492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:30.692269 10492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:30.693442 10492 hybrid_clock.cc:648] HybridClock initialized: now 1786502250693423 us; error 36 us; skew 500 ppm
Aug 12 02:37:30 dist-test-slave-rmhg krb5kdc[10321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502250, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:30.696306 10492 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:37:30.697098 10492 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:30.697211 10492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:30.697423 10492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:30.699749 10492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "6c743089cfd64feb85bcdbf225851c9d"
format_stamp: "Formatted at 2026-08-12 02:37:30 on dist-test-slave-rmhg"
server_key: "373daa56236def03c483f24b54cb6fab"
server_key_iv: "4357f30edf4feb6b7beddbbdd6dcb118"
server_key_version: "encryptionkey@0"
I20260812 02:37:30.700347 10492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "6c743089cfd64feb85bcdbf225851c9d"
format_stamp: "Formatted at 2026-08-12 02:37:30 on dist-test-slave-rmhg"
server_key: "373daa56236def03c483f24b54cb6fab"
server_key_iv: "4357f30edf4feb6b7beddbbdd6dcb118"
server_key_version: "encryptionkey@0"
I20260812 02:37:30.704348 10492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:37:30.706972 10519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:30.708290 10492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 02:37:30.708416 10492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "6c743089cfd64feb85bcdbf225851c9d"
format_stamp: "Formatted at 2026-08-12 02:37:30 on dist-test-slave-rmhg"
server_key: "373daa56236def03c483f24b54cb6fab"
server_key_iv: "4357f30edf4feb6b7beddbbdd6dcb118"
server_key_version: "encryptionkey@0"
I20260812 02:37:30.708528 10492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:30.730752 10492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:30.734118 10492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:30.734306 10492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:30.734933 10492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:30.735802 10492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:30.735849 10492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:30.735890 10492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:30.735921 10492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:30.745389 10492 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:34865
I20260812 02:37:30.745411 10723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:34865 every 8 connection(s)
I20260812 02:37:30.746057 10492 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:37:30.754822 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 10492
I20260812 02:37:30.754977 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:37:30.755329 3582 external_mini_cluster.cc:1428] Setting key 1d17807c0947c529eea9d8617ee14581
Aug 12 02:37:30 dist-test-slave-rmhg krb5kdc[10321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502250, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:30.760324 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:30.748094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:46499 (local address 127.3.127.190:36009)
0812 02:37:30.748248 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:30.748251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:30.749457 (+ 1206us) server_negotiation.cc:407] Connection header received
0812 02:37:30.750318 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:30.750321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:30.750371 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:30.750448 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:30.751671 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:30.752221 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:30.753381 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:30.753611 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:30.753700 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:30.756492 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:30.756495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:30.756497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:30.756523 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:30.758129 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:30.758696 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:30.758700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:30.758701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:30.758797 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:30.759132 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:30.759136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:30.759138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:30.759337 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:30.759429 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:30.759999 (+ 570us) server_negotiation.cc:299] Negotiation successful
0812 02:37:30.760158 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 02:37:30.761543 10726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:30.748444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36009 (local address 127.3.127.129:46499)
0812 02:37:30.749305 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:30.749351 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:30.750136 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:30.750608 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:30.750615 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:30.750985 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:30.751554 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:30.751569 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:30.752316 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:30.752318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:30.753284 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:30.753294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:30.753781 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:30.753785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:30.753945 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:30.754564 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:30.754587 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:30.756312 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:30.758266 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:30.758271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:30.758286 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:30.758600 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:30.758879 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:30.758881 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:30.758882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:30.759031 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:30.759442 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:30.759449 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:30.759753 (+ 304us) client_negotiation.cc:769] Sending connection context
0812 02:37:30.759936 (+ 183us) client_negotiation.cc:240] Negotiation successful
0812 02:37:30.760298 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":132,"threads_started":1}
I20260812 02:37:30.763203 10724 heartbeater.cc:347] Connected to a master server at 127.3.127.190:36009
I20260812 02:37:30.763491 10724 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:30.764065 10724 heartbeater.cc:511] Master 127.3.127.190:36009 requested a full tablet report, sending...
I20260812 02:37:30.765833 10391 ts_manager.cc:194] Registered new tserver with Master: 6c743089cfd64feb85bcdbf225851c9d (127.3.127.129:34865)
WARNING: no policy specified for kudu/127.3.127.130@KRBTEST.COM; defaulting to no policy
I20260812 02:37:30.767047 10391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:46499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
I20260812 02:37:30.826375 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:36009
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:41871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:30.943425 10729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:30.943742 10729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:30.943806 10729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:30.948014 10729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:30.948096 10729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:30.948181 10729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:37:30.953827 10729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:41871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:36009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:30.955018 10729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:30.956104 10729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:30.965907 10743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:30.965919 10740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:30.965907 10738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:30.966185 10729 server_base.cc:1034] running on GCE node
I20260812 02:37:30.966696 10729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:30.967379 10729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:30.968557 10729 hybrid_clock.cc:648] HybridClock initialized: now 1786502250968515 us; error 54 us; skew 500 ppm
Aug 12 02:37:30 dist-test-slave-rmhg krb5kdc[10321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502250, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:30.971599 10729 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:37:30.972446 10729 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:30.972517 10729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:30.972736 10729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:30.974931 10729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "31ed601633c24dc39a0950f0a2336842"
format_stamp: "Formatted at 2026-08-12 02:37:30 on dist-test-slave-rmhg"
server_key: "360ff3e69060efcf43ce0f74c83ef4f1"
server_key_iv: "aef6f8b8e98709954fc7995a2cd9fbe9"
server_key_version: "encryptionkey@0"
I20260812 02:37:30.975500 10729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "31ed601633c24dc39a0950f0a2336842"
format_stamp: "Formatted at 2026-08-12 02:37:30 on dist-test-slave-rmhg"
server_key: "360ff3e69060efcf43ce0f74c83ef4f1"
server_key_iv: "aef6f8b8e98709954fc7995a2cd9fbe9"
server_key_version: "encryptionkey@0"
I20260812 02:37:30.979583 10729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260812 02:37:30.982332 10749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:30.983727 10729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:30.983855 10729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "31ed601633c24dc39a0950f0a2336842"
format_stamp: "Formatted at 2026-08-12 02:37:30 on dist-test-slave-rmhg"
server_key: "360ff3e69060efcf43ce0f74c83ef4f1"
server_key_iv: "aef6f8b8e98709954fc7995a2cd9fbe9"
server_key_version: "encryptionkey@0"
I20260812 02:37:30.983963 10729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:30.995096 10729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:30.998328 10729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:30.998518 10729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:30.999126 10729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:30.999986 10729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:31.000036 10729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:31.000089 10729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:31.000120 10729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:31.009066 10729 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:35425
I20260812 02:37:31.009089 10925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:35425 every 8 connection(s)
I20260812 02:37:31.009785 10729 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:37:31.014048 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 10729
I20260812 02:37:31.014174 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:37:31.014510 3582 external_mini_cluster.cc:1428] Setting key 1c25d9ccba4ac5e569e4255ee214dedb
Aug 12 02:37:31 dist-test-slave-rmhg krb5kdc[10321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502250, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:31.024791 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.011644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:44535 (local address 127.3.127.190:36009)
0812 02:37:31.011781 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:31.011785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:31.013169 (+ 1384us) server_negotiation.cc:407] Connection header received
0812 02:37:31.014213 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:31.014217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:31.014277 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:31.014352 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:31.015861 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.016531 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.017796 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.018104 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.018216 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:31.021078 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:31.021081 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:31.021083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:31.021109 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:31.022545 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:31.023182 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:31.023185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:31.023186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:31.023259 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:31.023626 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:31.023628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:31.023629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:31.023807 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:31.023878 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:31.024551 (+ 673us) server_negotiation.cc:299] Negotiation successful
0812 02:37:31.024662 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.3.127.131@KRBTEST.COM; defaulting to no policy
I20260812 02:37:31.026124 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.012076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36009 (local address 127.3.127.130:44535)
0812 02:37:31.012992 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:31.013039 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:31.013928 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:31.014574 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:31.014585 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:31.015058 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:31.015717 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:31.015739 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.016684 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.016687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:31.017677 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:31.017688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.018278 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.018282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:31.018486 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:31.019106 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:31.019127 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:31.020893 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:31.022681 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:31.022688 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:31.022706 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:31.023083 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:31.023340 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:31.023343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:31.023345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:31.023539 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:31.023907 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:31.023914 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:31.024278 (+ 364us) client_negotiation.cc:769] Sending connection context
0812 02:37:31.024532 (+ 254us) client_negotiation.cc:240] Negotiation successful
0812 02:37:31.024918 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":116,"threads_started":1}
I20260812 02:37:31.027431 10926 heartbeater.cc:347] Connected to a master server at 127.3.127.190:36009
I20260812 02:37:31.027733 10926 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:31.028230 10926 heartbeater.cc:511] Master 127.3.127.190:36009 requested a full tablet report, sending...
I20260812 02:37:31.029335 10391 ts_manager.cc:194] Registered new tserver with Master: 31ed601633c24dc39a0950f0a2336842 (127.3.127.130:35425)
I20260812 02:37:31.029935 10391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:44535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
I20260812 02:37:31.071949 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:36009
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:41871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:31.192303 10938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:31.192622 10938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:31.192677 10938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:31.196765 10938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:31.196837 10938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:31.196900 10938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:37:31.202368 10938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:41871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:36009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:31.203485 10938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:31.204517 10938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:31.214768 10953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:31.214748 10947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:31.214752 10948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:31.214809 10938 server_base.cc:1034] running on GCE node
I20260812 02:37:31.215530 10938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:31.216329 10938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:31.217535 10938 hybrid_clock.cc:648] HybridClock initialized: now 1786502251217492 us; error 57 us; skew 500 ppm
Aug 12 02:37:31 dist-test-slave-rmhg krb5kdc[10321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502251, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:31.220721 10938 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:37:31.221724 10938 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:31.221808 10938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:31.222033 10938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:31.224308 10938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "df02f7352fae4b7aaaadbd63c86d99a5"
format_stamp: "Formatted at 2026-08-12 02:37:31 on dist-test-slave-rmhg"
server_key: "84cfbef91eed5f511e5884ad50b64c56"
server_key_iv: "ad487382ea59fc0a66307316dbce4e23"
server_key_version: "encryptionkey@0"
I20260812 02:37:31.224916 10938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "df02f7352fae4b7aaaadbd63c86d99a5"
format_stamp: "Formatted at 2026-08-12 02:37:31 on dist-test-slave-rmhg"
server_key: "84cfbef91eed5f511e5884ad50b64c56"
server_key_iv: "ad487382ea59fc0a66307316dbce4e23"
server_key_version: "encryptionkey@0"
I20260812 02:37:31.229264 10938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:37:31.232138 10963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:31.233664 10938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:31.233809 10938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "df02f7352fae4b7aaaadbd63c86d99a5"
format_stamp: "Formatted at 2026-08-12 02:37:31 on dist-test-slave-rmhg"
server_key: "84cfbef91eed5f511e5884ad50b64c56"
server_key_iv: "ad487382ea59fc0a66307316dbce4e23"
server_key_version: "encryptionkey@0"
I20260812 02:37:31.233934 10938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:31.257688 10938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:31.261368 10938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:31.261667 10938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:31.262390 10938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:31.263420 10938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:31.263473 10938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:31.263530 10938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:31.263558 10938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:31.273532 10938 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:39347
I20260812 02:37:31.273555 11138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:39347 every 8 connection(s)
I20260812 02:37:31.274212 10938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:37:31.280395 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 10938
I20260812 02:37:31.280548 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:37:31.280895 3582 external_mini_cluster.cc:1428] Setting key aee594d334c7757b3472ae877a9c667c
Aug 12 02:37:31 dist-test-slave-rmhg krb5kdc[10321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502251, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:31.291917 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.276383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:49283 (local address 127.3.127.190:36009)
0812 02:37:31.276580 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:31.276587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:31.278228 (+ 1641us) server_negotiation.cc:407] Connection header received
0812 02:37:31.279215 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:31.279219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:31.279277 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:31.279364 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:31.281332 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.282059 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.284189 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.284478 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.284581 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:31.287830 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:31.287839 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:31.287842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:31.287878 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:31.289286 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:31.289894 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:31.289898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:31.289900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:31.289984 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:31.290398 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:31.290401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:31.290403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:31.290669 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:31.290765 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:31.291555 (+ 790us) server_negotiation.cc:299] Negotiation successful
0812 02:37:31.291707 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 02:37:31.292855 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.277035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36009 (local address 127.3.127.131:49283)
0812 02:37:31.278003 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:31.278060 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:31.278928 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:31.279638 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:31.279646 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:31.280182 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:31.281060 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:31.281086 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.282214 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.282217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:31.284030 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:31.284045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.284672 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.284678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:31.284888 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:31.285595 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:31.285633 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:31.287573 (+ 1940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:31.289405 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:31.289413 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:31.289433 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:31.289768 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:31.290083 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:31.290087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:31.290089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:31.290277 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:31.290790 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:31.290798 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:31.291106 (+ 308us) client_negotiation.cc:769] Sending connection context
0812 02:37:31.291477 (+ 371us) client_negotiation.cc:240] Negotiation successful
0812 02:37:31.291922 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":140,"threads_started":1}
I20260812 02:37:31.294342 11141 heartbeater.cc:347] Connected to a master server at 127.3.127.190:36009
I20260812 02:37:31.294697 11141 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:31.295291 11141 heartbeater.cc:511] Master 127.3.127.190:36009 requested a full tablet report, sending...
I20260812 02:37:31.296711 10391 ts_manager.cc:194] Registered new tserver with Master: df02f7352fae4b7aaaadbd63c86d99a5 (127.3.127.131:39347)
I20260812 02:37:31.297380 10391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:49283
I20260812 02:37:31.308310 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:37:31.318603 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.310540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35088 (local address 127.3.127.190:36009)
0812 02:37:31.310680 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:31.310684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:31.310868 (+ 184us) server_negotiation.cc:407] Connection header received
0812 02:37:31.310991 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:31.310994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:31.311042 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:31.311122 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:31.312000 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.312557 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.313636 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.313888 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.314003 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:31.315485 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:31.315488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:31.315490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:31.315523 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:31.316872 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:31.317409 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:31.317412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:31.317413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:31.317499 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:31.317737 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:31.317739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:31.317740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:31.317991 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:31.318097 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:31.318357 (+ 260us) server_negotiation.cc:299] Negotiation successful
0812 02:37:31.318468 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 02:37:31.322057 10391 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:37:31.325393 10391 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:37:31.342528 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.335923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56388 (local address 127.3.127.129:34865)
0812 02:37:31.337373 (+ 1450us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:31.337381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:31.337408 (+ 27us) server_negotiation.cc:407] Connection header received
0812 02:37:31.337512 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:31.337517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:31.337719 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:31.337882 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:31.338383 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.339372 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.340995 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.341750 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.341863 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:31.342057 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:31.342231 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 02:37:31.342356 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1329,"thread_start_us":527,"threads_started":1}
I20260812 02:37:31.342724 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.335814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:35425 (local address 127.0.0.1:50044)
0812 02:37:31.336699 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:31.336712 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:31.336805 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:31.338021 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:31.338024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:31.338046 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:31.338295 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:31.338304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.339784 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.339787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:31.341023 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:31.341035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.342273 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.342277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:31.342371 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:31.342398 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 02:37:31.342444 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 02:37:31.342517 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":118,"thread_start_us":639,"threads_started":1}
I20260812 02:37:31.342752 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.336580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:39347 (local address 127.0.0.1:46432)
0812 02:37:31.336998 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:31.337011 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:31.337090 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:31.338036 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:31.338038 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:31.338052 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:31.338422 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:31.338428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.340075 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.340079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:31.341280 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:31.341289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.342393 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.342396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:31.342473 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:31.342491 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 02:37:31.342530 (+ 39us) client_negotiation.cc:240] Negotiation successful
0812 02:37:31.342585 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":224,"thread_start_us":121,"threads_started":1}
I20260812 02:37:31.343098 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.336954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50044 (local address 127.3.127.130:35425)
0812 02:37:31.337456 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:31.337469 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:31.337495 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:37:31.337579 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:31.337585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:31.337813 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:31.338225 (+ 412us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:31.338393 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.339617 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.341255 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.342115 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.342604 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:31.342781 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:31.342841 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 02:37:31.342942 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":121,"threads_started":1}
I20260812 02:37:31.342519 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.335736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:34865 (local address 127.0.0.1:56388)
0812 02:37:31.336186 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:31.336245 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:31.336380 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:31.337895 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:31.337901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:31.337927 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:31.338255 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:31.338268 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.339523 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.339526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:31.340860 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:31.340873 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.341892 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.341896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:31.342016 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:31.342057 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 02:37:31.342183 (+ 126us) client_negotiation.cc:240] Negotiation successful
0812 02:37:31.342356 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":108,"threads_started":1}
I20260812 02:37:31.343979 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.337065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.3.127.131:39347)
0812 02:37:31.337597 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:31.337608 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:31.337627 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:37:31.337694 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:31.337698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:31.337858 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:31.338585 (+ 727us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:31.338755 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.339902 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.341462 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.342273 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.343493 (+ 1220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:31.343658 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:31.343712 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 02:37:31.343821 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":248,"threads_started":1}
I20260812 02:37:31.346276 10610 tablet_service.cc:1467] Processing CreateTablet for tablet 38c76193f83942a6a126fdf48ea36baf (DEFAULT_TABLE table=test-table [id=dcc288c6bfe346ababf0976e95880985]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:31.346325 10829 tablet_service.cc:1467] Processing CreateTablet for tablet 38c76193f83942a6a126fdf48ea36baf (DEFAULT_TABLE table=test-table [id=dcc288c6bfe346ababf0976e95880985]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:31.347457 10829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c76193f83942a6a126fdf48ea36baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:31.347457 10610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c76193f83942a6a126fdf48ea36baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:31.347115 11036 tablet_service.cc:1467] Processing CreateTablet for tablet 38c76193f83942a6a126fdf48ea36baf (DEFAULT_TABLE table=test-table [id=dcc288c6bfe346ababf0976e95880985]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:31.348198 11036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c76193f83942a6a126fdf48ea36baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:31.354965 11168 tablet_bootstrap.cc:492] T 38c76193f83942a6a126fdf48ea36baf P 6c743089cfd64feb85bcdbf225851c9d: Bootstrap starting.
I20260812 02:37:31.356482 11169 tablet_bootstrap.cc:492] T 38c76193f83942a6a126fdf48ea36baf P 31ed601633c24dc39a0950f0a2336842: Bootstrap starting.
I20260812 02:37:31.356482 11170 tablet_bootstrap.cc:492] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5: Bootstrap starting.
I20260812 02:37:31.357573 11168 tablet_bootstrap.cc:654] T 38c76193f83942a6a126fdf48ea36baf P 6c743089cfd64feb85bcdbf225851c9d: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:31.358503 11168 log.cc:824] T 38c76193f83942a6a126fdf48ea36baf P 6c743089cfd64feb85bcdbf225851c9d: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:31.359022 11170 tablet_bootstrap.cc:654] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:31.359022 11169 tablet_bootstrap.cc:654] T 38c76193f83942a6a126fdf48ea36baf P 31ed601633c24dc39a0950f0a2336842: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:31.360263 11169 log.cc:824] T 38c76193f83942a6a126fdf48ea36baf P 31ed601633c24dc39a0950f0a2336842: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:31.361073 11168 tablet_bootstrap.cc:492] T 38c76193f83942a6a126fdf48ea36baf P 6c743089cfd64feb85bcdbf225851c9d: No bootstrap required, opened a new log
I20260812 02:37:31.361455 11168 ts_tablet_manager.cc:1397] T 38c76193f83942a6a126fdf48ea36baf P 6c743089cfd64feb85bcdbf225851c9d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20260812 02:37:31.362286 11170 log.cc:824] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:31.363134 11169 tablet_bootstrap.cc:492] T 38c76193f83942a6a126fdf48ea36baf P 31ed601633c24dc39a0950f0a2336842: No bootstrap required, opened a new log
I20260812 02:37:31.363430 11169 ts_tablet_manager.cc:1397] T 38c76193f83942a6a126fdf48ea36baf P 31ed601633c24dc39a0950f0a2336842: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 02:37:31.364360 11170 tablet_bootstrap.cc:492] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5: No bootstrap required, opened a new log
I20260812 02:37:31.364580 11170 ts_tablet_manager.cc:1397] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260812 02:37:31.366693 11168 raft_consensus.cc:348] T 38c76193f83942a6a126fdf48ea36baf P 6c743089cfd64feb85bcdbf225851c9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c743089cfd64feb85bcdbf225851c9d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34865 } } peers { permanent_uuid: "31ed601633c24dc39a0950f0a2336842" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35425 } } peers { permanent_uuid: "df02f7352fae4b7aaaadbd63c86d99a5" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39347 } }
I20260812 02:37:31.367019 11168 raft_consensus.cc:374] T 38c76193f83942a6a126fdf48ea36baf P 6c743089cfd64feb85bcdbf225851c9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:31.367164 11168 raft_consensus.cc:729] T 38c76193f83942a6a126fdf48ea36baf P 6c743089cfd64feb85bcdbf225851c9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c743089cfd64feb85bcdbf225851c9d, State: Initialized, Role: FOLLOWER
I20260812 02:37:31.367692 11168 consensus_queue.cc:260] T 38c76193f83942a6a126fdf48ea36baf P 6c743089cfd64feb85bcdbf225851c9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c743089cfd64feb85bcdbf225851c9d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34865 } } peers { permanent_uuid: "31ed601633c24dc39a0950f0a2336842" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35425 } } peers { permanent_uuid: "df02f7352fae4b7aaaadbd63c86d99a5" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39347 } }
I20260812 02:37:31.367568 11169 raft_consensus.cc:348] T 38c76193f83942a6a126fdf48ea36baf P 31ed601633c24dc39a0950f0a2336842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c743089cfd64feb85bcdbf225851c9d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34865 } } peers { permanent_uuid: "31ed601633c24dc39a0950f0a2336842" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35425 } } peers { permanent_uuid: "df02f7352fae4b7aaaadbd63c86d99a5" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39347 } }
I20260812 02:37:31.367872 11169 raft_consensus.cc:374] T 38c76193f83942a6a126fdf48ea36baf P 31ed601633c24dc39a0950f0a2336842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:31.367959 11169 raft_consensus.cc:729] T 38c76193f83942a6a126fdf48ea36baf P 31ed601633c24dc39a0950f0a2336842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31ed601633c24dc39a0950f0a2336842, State: Initialized, Role: FOLLOWER
I20260812 02:37:31.368057 11170 raft_consensus.cc:348] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c743089cfd64feb85bcdbf225851c9d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34865 } } peers { permanent_uuid: "31ed601633c24dc39a0950f0a2336842" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35425 } } peers { permanent_uuid: "df02f7352fae4b7aaaadbd63c86d99a5" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39347 } }
I20260812 02:37:31.368383 11170 raft_consensus.cc:374] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:31.368481 11170 raft_consensus.cc:729] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df02f7352fae4b7aaaadbd63c86d99a5, State: Initialized, Role: FOLLOWER
I20260812 02:37:31.368490 11169 consensus_queue.cc:260] T 38c76193f83942a6a126fdf48ea36baf P 31ed601633c24dc39a0950f0a2336842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c743089cfd64feb85bcdbf225851c9d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34865 } } peers { permanent_uuid: "31ed601633c24dc39a0950f0a2336842" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35425 } } peers { permanent_uuid: "df02f7352fae4b7aaaadbd63c86d99a5" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39347 } }
I20260812 02:37:31.368568 10724 heartbeater.cc:503] Master 127.3.127.190:36009 was elected leader, sending a full tablet report...
I20260812 02:37:31.368964 11168 ts_tablet_manager.cc:1428] T 38c76193f83942a6a126fdf48ea36baf P 6c743089cfd64feb85bcdbf225851c9d: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260812 02:37:31.368970 11170 consensus_queue.cc:260] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c743089cfd64feb85bcdbf225851c9d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34865 } } peers { permanent_uuid: "31ed601633c24dc39a0950f0a2336842" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35425 } } peers { permanent_uuid: "df02f7352fae4b7aaaadbd63c86d99a5" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39347 } }
I20260812 02:37:31.369745 11169 ts_tablet_manager.cc:1428] T 38c76193f83942a6a126fdf48ea36baf P 31ed601633c24dc39a0950f0a2336842: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 02:37:31.369969 11170 ts_tablet_manager.cc:1428] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260812 02:37:31.370217 10926 heartbeater.cc:503] Master 127.3.127.190:36009 was elected leader, sending a full tablet report...
I20260812 02:37:31.371277 11141 heartbeater.cc:503] Master 127.3.127.190:36009 was elected leader, sending a full tablet report...
I20260812 02:37:31.425021 11179 raft_consensus.cc:482] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:37:31.425308 11179 raft_consensus.cc:504] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c743089cfd64feb85bcdbf225851c9d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34865 } } peers { permanent_uuid: "31ed601633c24dc39a0950f0a2336842" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35425 } } peers { permanent_uuid: "df02f7352fae4b7aaaadbd63c86d99a5" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39347 } }
I20260812 02:37:31.426537 11179 leader_election.cc:288] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c743089cfd64feb85bcdbf225851c9d (127.3.127.129:34865), 31ed601633c24dc39a0950f0a2336842 (127.3.127.130:35425)
I20260812 02:37:31.430805 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.426984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:56587 (local address 127.3.127.130:35425)
0812 02:37:31.427150 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:31.427155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:31.427579 (+ 424us) server_negotiation.cc:407] Connection header received
0812 02:37:31.427642 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:31.427645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:31.427692 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:31.427782 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:31.428080 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.428652 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.429787 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.430202 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.430319 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:31.430377 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:31.430596 (+ 219us) server_negotiation.cc:299] Negotiation successful
0812 02:37:31.430674 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 02:37:31.430809 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.427153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:46129 (local address 127.3.127.129:34865)
0812 02:37:31.427288 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:31.427293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:31.427311 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:37:31.427371 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:31.427375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:31.427431 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:31.427509 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:31.427894 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.428454 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.429787 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.430208 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.430318 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:31.430377 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:31.430620 (+ 243us) server_negotiation.cc:299] Negotiation successful
0812 02:37:31.430707 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 02:37:31.430805 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.426864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:34865 (local address 127.3.127.131:46129)
0812 02:37:31.427014 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:31.427031 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:31.427156 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:31.427613 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:31.427616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:31.427640 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:31.427777 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:31.427785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.428560 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.428563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:31.429650 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:31.429661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.430325 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.430328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:31.430487 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:31.430512 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 02:37:31.430585 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 02:37:31.430683 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":105}
I20260812 02:37:31.430819 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.426864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:35425 (local address 127.3.127.131:56587)
0812 02:37:31.427389 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:31.427401 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:31.427508 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:31.427812 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:31.427814 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:31.427830 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:31.427950 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:31.427956 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.428760 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.428763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:31.429653 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:31.429663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.430319 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.430322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:31.430474 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:31.430495 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 02:37:31.430554 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 02:37:31.430647 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":97,"thread_start_us":133,"threads_started":1}
I20260812 02:37:31.431598 10648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c76193f83942a6a126fdf48ea36baf" candidate_uuid: "df02f7352fae4b7aaaadbd63c86d99a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c743089cfd64feb85bcdbf225851c9d" is_pre_election: true
I20260812 02:37:31.431694 10859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c76193f83942a6a126fdf48ea36baf" candidate_uuid: "df02f7352fae4b7aaaadbd63c86d99a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31ed601633c24dc39a0950f0a2336842" is_pre_election: true
I20260812 02:37:31.431941 10648 raft_consensus.cc:2454] T 38c76193f83942a6a126fdf48ea36baf P 6c743089cfd64feb85bcdbf225851c9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df02f7352fae4b7aaaadbd63c86d99a5 in term 0.
I20260812 02:37:31.432003 10859 raft_consensus.cc:2454] T 38c76193f83942a6a126fdf48ea36baf P 31ed601633c24dc39a0950f0a2336842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df02f7352fae4b7aaaadbd63c86d99a5 in term 0.
I20260812 02:37:31.432534 10971 leader_election.cc:302] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31ed601633c24dc39a0950f0a2336842, df02f7352fae4b7aaaadbd63c86d99a5; no voters:
I20260812 02:37:31.432838 11179 raft_consensus.cc:2781] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:37:31.432925 11179 raft_consensus.cc:482] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:37:31.432955 11179 raft_consensus.cc:3037] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:31.484078 11179 raft_consensus.cc:504] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c743089cfd64feb85bcdbf225851c9d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34865 } } peers { permanent_uuid: "31ed601633c24dc39a0950f0a2336842" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35425 } } peers { permanent_uuid: "df02f7352fae4b7aaaadbd63c86d99a5" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39347 } }
I20260812 02:37:31.484681 11179 leader_election.cc:288] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 [CANDIDATE]: Term 1 election: Requested vote from peers 6c743089cfd64feb85bcdbf225851c9d (127.3.127.129:34865), 31ed601633c24dc39a0950f0a2336842 (127.3.127.130:35425)
I20260812 02:37:31.485314 10648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c76193f83942a6a126fdf48ea36baf" candidate_uuid: "df02f7352fae4b7aaaadbd63c86d99a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c743089cfd64feb85bcdbf225851c9d"
I20260812 02:37:31.485337 10859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c76193f83942a6a126fdf48ea36baf" candidate_uuid: "df02f7352fae4b7aaaadbd63c86d99a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31ed601633c24dc39a0950f0a2336842"
I20260812 02:37:31.485518 10648 raft_consensus.cc:3037] T 38c76193f83942a6a126fdf48ea36baf P 6c743089cfd64feb85bcdbf225851c9d [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:31.485581 10859 raft_consensus.cc:3037] T 38c76193f83942a6a126fdf48ea36baf P 31ed601633c24dc39a0950f0a2336842 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:31.487135 10859 raft_consensus.cc:2454] T 38c76193f83942a6a126fdf48ea36baf P 31ed601633c24dc39a0950f0a2336842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df02f7352fae4b7aaaadbd63c86d99a5 in term 1.
I20260812 02:37:31.487215 10648 raft_consensus.cc:2454] T 38c76193f83942a6a126fdf48ea36baf P 6c743089cfd64feb85bcdbf225851c9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df02f7352fae4b7aaaadbd63c86d99a5 in term 1.
I20260812 02:37:31.487605 10971 leader_election.cc:302] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31ed601633c24dc39a0950f0a2336842, df02f7352fae4b7aaaadbd63c86d99a5; no voters:
I20260812 02:37:31.487860 11179 raft_consensus.cc:2781] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:31.488369 11179 raft_consensus.cc:686] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 [term 1 LEADER]: Becoming Leader. State: Replica: df02f7352fae4b7aaaadbd63c86d99a5, State: Running, Role: LEADER
I20260812 02:37:31.488816 11179 consensus_queue.cc:237] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c743089cfd64feb85bcdbf225851c9d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34865 } } peers { permanent_uuid: "31ed601633c24dc39a0950f0a2336842" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35425 } } peers { permanent_uuid: "df02f7352fae4b7aaaadbd63c86d99a5" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39347 } }
I20260812 02:37:31.492501 10391 catalog_manager.cc:5495] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to df02f7352fae4b7aaaadbd63c86d99a5 (127.3.127.131). New cstate: current_term: 1 leader_uuid: "df02f7352fae4b7aaaadbd63c86d99a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c743089cfd64feb85bcdbf225851c9d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31ed601633c24dc39a0950f0a2336842" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df02f7352fae4b7aaaadbd63c86d99a5" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39347 } health_report { overall_health: HEALTHY } } }
W20260812 02:37:31.497438 10725 tablet.cc:2369] T 38c76193f83942a6a126fdf48ea36baf P 6c743089cfd64feb85bcdbf225851c9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:37:31.511041 10929 tablet.cc:2369] T 38c76193f83942a6a126fdf48ea36baf P 31ed601633c24dc39a0950f0a2336842: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:37:31.525285 11142 tablet.cc:2369] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:37:31.535836 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.532178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46448 (local address 127.3.127.131:39347)
0812 02:37:31.532333 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:31.532337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:31.532353 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:37:31.532466 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:31.532469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:31.532527 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:31.532668 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:31.532926 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.533481 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.534333 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:31.534554 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:31.534643 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:31.534951 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:31.535417 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:31.535541 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:31.535599 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 02:37:31.535665 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 02:37:31.542631 10648 raft_consensus.cc:1261] T 38c76193f83942a6a126fdf48ea36baf P 6c743089cfd64feb85bcdbf225851c9d [term 1 FOLLOWER]: Refusing update from remote peer df02f7352fae4b7aaaadbd63c86d99a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:31.542631 10859 raft_consensus.cc:1261] T 38c76193f83942a6a126fdf48ea36baf P 31ed601633c24dc39a0950f0a2336842 [term 1 FOLLOWER]: Refusing update from remote peer df02f7352fae4b7aaaadbd63c86d99a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:31.543454 11182 consensus_queue.cc:1035] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ed601633c24dc39a0950f0a2336842" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:31.543716 11179 consensus_queue.cc:1035] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c743089cfd64feb85bcdbf225851c9d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:31.555006 11190 mvcc.cc:205] Tried to move back new op lower bound from 7317513222310375424 to 7317513222102011904. Current Snapshot: MvccSnapshot[applied={T|T < 7317513222310375424}]
I20260812 02:37:31.555224 11188 mvcc.cc:205] Tried to move back new op lower bound from 7317513222310375424 to 7317513222102011904. Current Snapshot: MvccSnapshot[applied={T|T < 7317513222310375424 or (T in {7317513222310375424})}]
I20260812 02:37:31.562952 10389 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:31.563149 10389 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:31.565907 10389 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5f8c0e7a827457a953245163fde5735: Sending DeleteTablet for 3 replicas of tablet 38c76193f83942a6a126fdf48ea36baf
I20260812 02:37:31.566879 11036 tablet_service.cc:1514] Processing DeleteTablet for tablet 38c76193f83942a6a126fdf48ea36baf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:31 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:46432
I20260812 02:37:31.566922 10829 tablet_service.cc:1514] Processing DeleteTablet for tablet 38c76193f83942a6a126fdf48ea36baf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:31 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:50044
I20260812 02:37:31.566896 10610 tablet_service.cc:1514] Processing DeleteTablet for tablet 38c76193f83942a6a126fdf48ea36baf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:31 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:56388
I20260812 02:37:31.567500 11206 tablet_replica.cc:330] T 38c76193f83942a6a126fdf48ea36baf P 31ed601633c24dc39a0950f0a2336842: stopping tablet replica
I20260812 02:37:31.567806 11205 tablet_replica.cc:330] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5: stopping tablet replica
I20260812 02:37:31.567909 11206 raft_consensus.cc:2229] T 38c76193f83942a6a126fdf48ea36baf P 31ed601633c24dc39a0950f0a2336842 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:31.568243 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 10492
I20260812 02:37:31.568312 11206 raft_consensus.cc:2258] T 38c76193f83942a6a126fdf48ea36baf P 31ed601633c24dc39a0950f0a2336842 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:31.568248 11205 raft_consensus.cc:2229] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:37:31.569257 11205 raft_consensus.cc:2258] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:31.573206 11205 ts_tablet_manager.cc:1905] T 38c76193f83942a6a126fdf48ea36baf P df02f7352fae4b7aaaadbd63c86d99a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:31.574782 11206 ts_tablet_manager.cc:1905] T 38c76193f83942a6a126fdf48ea36baf P 31ed601633c24dc39a0950f0a2336842: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 02:37:31.580034 10363 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.127.129:34865 (error 108)
I20260812 02:37:31.581041 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 10729
I20260812 02:37:31.581362 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.580975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:34865 (local address 127.0.0.1:56396)
0812 02:37:31.581174 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:31.581257 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.129:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260812 02:37:31.593454 10363 catalog_manager.cc:4584] TS 6c743089cfd64feb85bcdbf225851c9d (127.3.127.129:34865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38c76193f83942a6a126fdf48ea36baf: Network error: Client connection negotiation failed: client connection to 127.3.127.129:34865: connect: Connection refused (error 111)
I20260812 02:37:31.600814 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 10938
I20260812 02:37:31.608829 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.601214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:35425 (local address 127.0.0.1:50048)
0812 02:37:31.608540 (+ 7326us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:31.608675 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.130:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7169}
I20260812 02:37:31.646997 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.646585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:34865 (local address 127.0.0.1:56404)
0812 02:37:31.646787 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:31.646872 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.129:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 02:37:31.648083 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.647791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:34865 (local address 127.0.0.1:56420)
0812 02:37:31.647930 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:31.647998 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.129:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 02:37:31.653077 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.652775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:35425 (local address 127.0.0.1:50062)
0812 02:37:31.652930 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:31.652996 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.130:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 02:37:31.653914 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.653695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:35425 (local address 127.0.0.1:50078)
0812 02:37:31.653801 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:31.653851 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.130:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 02:37:31.694693 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 10344
2026-08-12T02:37:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2828 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T02:37:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:37:31Z Disabled control of system clock
I20260812 02:37:31.746308 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:38147
--webserver_interface=127.3.127.190
--webserver_port=0
--builtin_ntp_servers=127.3.127.148:34061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:38147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 02:37:31.869527 11214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:31.869874 11214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:31.869932 11214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:31.874114 11214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:31.874198 11214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:31.874219 11214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:31.874240 11214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:31.874259 11214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:37:31.879307 11214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:34061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:38147
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:38147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:31.880548 11214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:31.882079 11214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:31.891274 11224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:31.891325 11221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:31.891324 11228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:31.891521 11214 server_base.cc:1034] running on GCE node
I20260812 02:37:31.892264 11214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:31.893549 11214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:31.894729 11214 hybrid_clock.cc:648] HybridClock initialized: now 1786502251894699 us; error 50 us; skew 500 ppm
I20260812 02:37:31.896835 11214 webserver.cc:460] Webserver started at http://127.3.127.190:46055/ using document root <none> and password file <none>
I20260812 02:37:31.897526 11214 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:31.897604 11214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:31.897822 11214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:31.899963 11214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "84c34cc6d9254609bc7d073d9247e7dc"
format_stamp: "Formatted at 2026-08-12 02:37:31 on dist-test-slave-rmhg"
server_key: "08fa4590a007c99d01736f210ea09e1d"
server_key_iv: "6be584fb52177c2a4b9b695f848c36d6"
server_key_version: "encryptionkey@0"
I20260812 02:37:31.900533 11214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "84c34cc6d9254609bc7d073d9247e7dc"
format_stamp: "Formatted at 2026-08-12 02:37:31 on dist-test-slave-rmhg"
server_key: "08fa4590a007c99d01736f210ea09e1d"
server_key_iv: "6be584fb52177c2a4b9b695f848c36d6"
server_key_version: "encryptionkey@0"
I20260812 02:37:31.904551 11214 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:37:31.907269 11235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:31.908859 11214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:37:31.909026 11214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "84c34cc6d9254609bc7d073d9247e7dc"
format_stamp: "Formatted at 2026-08-12 02:37:31 on dist-test-slave-rmhg"
server_key: "08fa4590a007c99d01736f210ea09e1d"
server_key_iv: "6be584fb52177c2a4b9b695f848c36d6"
server_key_version: "encryptionkey@0"
I20260812 02:37:31.909196 11214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:31.914296 11214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:31.914987 11214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:31.915172 11214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:31.922538 11214 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:38147
I20260812 02:37:31.922561 11328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:38147 every 8 connection(s)
I20260812 02:37:31.923547 11214 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:37:31.927220 11329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:31.933408 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 11214
I20260812 02:37:31.933547 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:37:31.933732 11329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc: Bootstrap starting.
I20260812 02:37:31.933861 3582 external_mini_cluster.cc:1428] Setting key 22d06fba8a2de3b72b59450b248ab437
I20260812 02:37:31.936645 11329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:31.937489 11329 log.cc:824] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:31.939692 11329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc: No bootstrap required, opened a new log
I20260812 02:37:31.939947 11334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:31.935448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.3.127.190:38147)
0812 02:37:31.936257 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:31.936276 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:31.936318 (+ 42us) server_negotiation.cc:407] Connection header received
0812 02:37:31.937013 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:31.937034 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:31.937465 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:31.937833 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:31.938615 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:31.938619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:31.938633 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:37:31.938644 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:31.938703 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:37:31.938721 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:31.938820 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:31.938937 (+ 117us) server_negotiation.cc:299] Negotiation successful
0812 02:37:31.939206 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":159,"threads_started":1}
I20260812 02:37:31.943524 11329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c34cc6d9254609bc7d073d9247e7dc" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38147 } }
I20260812 02:37:31.943787 11329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:31.943837 11329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c34cc6d9254609bc7d073d9247e7dc, State: Initialized, Role: FOLLOWER
I20260812 02:37:31.944257 11329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c34cc6d9254609bc7d073d9247e7dc" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38147 } }
I20260812 02:37:31.944407 11329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:37:31.944490 11329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:37:31.944600 11329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:31.990716 11329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c34cc6d9254609bc7d073d9247e7dc" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38147 } }
I20260812 02:37:31.991353 11329 leader_election.cc:302] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84c34cc6d9254609bc7d073d9247e7dc; no voters:
I20260812 02:37:31.991813 11329 leader_election.cc:288] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:37:31.991951 11336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:31.992246 11336 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc [term 1 LEADER]: Becoming Leader. State: Replica: 84c34cc6d9254609bc7d073d9247e7dc, State: Running, Role: LEADER
I20260812 02:37:31.992623 11336 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c34cc6d9254609bc7d073d9247e7dc" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38147 } }
I20260812 02:37:31.993394 11329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:37:31.994427 11339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84c34cc6d9254609bc7d073d9247e7dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c34cc6d9254609bc7d073d9247e7dc" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38147 } } }
I20260812 02:37:31.994580 11339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:31.995055 11340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84c34cc6d9254609bc7d073d9247e7dc. Latest consensus state: current_term: 1 leader_uuid: "84c34cc6d9254609bc7d073d9247e7dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c34cc6d9254609bc7d073d9247e7dc" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38147 } } }
I20260812 02:37:31.995241 11340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:31.995754 11349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:37:31.999500 11349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:37:32.005817 11349 catalog_manager.cc:1347] Generated new cluster ID: b65b8bdbd5be4dc19a3cf791f67c78c0
I20260812 02:37:32.005904 11349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:37:32.018968 11349 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:37:32.019374 11349 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:37:32.025746 11349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc: Generated new TSK 0
I20260812 02:37:32.026628 11349 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:37:32.032121 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38147
--builtin_ntp_servers=127.3.127.148:34061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 02:37:32.179868 11363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:32.180197 11363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:32.180243 11363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:32.184391 11363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:32.184484 11363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:32.184544 11363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:37:32.190824 11363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:34061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:32.192528 11363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:32.194554 11363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:32.205876 11371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:32.205897 11373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:32.205893 11378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:32.206285 11363 server_base.cc:1034] running on GCE node
I20260812 02:37:32.206885 11363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:32.207686 11363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:32.208887 11363 hybrid_clock.cc:648] HybridClock initialized: now 1786502252208841 us; error 67 us; skew 500 ppm
I20260812 02:37:32.211615 11363 webserver.cc:460] Webserver started at http://127.3.127.129:32805/ using document root <none> and password file <none>
I20260812 02:37:32.212335 11363 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:32.212416 11363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:32.212663 11363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:32.215348 11363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "0037954a57bf4e0ab82c704b5a229a0e"
format_stamp: "Formatted at 2026-08-12 02:37:32 on dist-test-slave-rmhg"
server_key: "32ce0d2dcd445ca486e0ad3d48cd296e"
server_key_iv: "db10002c066b4a37f6c3bc2baeb9bc45"
server_key_version: "encryptionkey@0"
I20260812 02:37:32.216050 11363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "0037954a57bf4e0ab82c704b5a229a0e"
format_stamp: "Formatted at 2026-08-12 02:37:32 on dist-test-slave-rmhg"
server_key: "32ce0d2dcd445ca486e0ad3d48cd296e"
server_key_iv: "db10002c066b4a37f6c3bc2baeb9bc45"
server_key_version: "encryptionkey@0"
I20260812 02:37:32.220649 11363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260812 02:37:32.224035 11391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:32.225927 11363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:32.226111 11363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "0037954a57bf4e0ab82c704b5a229a0e"
format_stamp: "Formatted at 2026-08-12 02:37:32 on dist-test-slave-rmhg"
server_key: "32ce0d2dcd445ca486e0ad3d48cd296e"
server_key_iv: "db10002c066b4a37f6c3bc2baeb9bc45"
server_key_version: "encryptionkey@0"
I20260812 02:37:32.226274 11363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:32.231902 11363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:32.232704 11363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:32.232909 11363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:32.233656 11363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:32.234714 11363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:32.234771 11363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:32.234819 11363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:32.234839 11363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:32.244406 11363 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:46077
I20260812 02:37:32.244457 11564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:46077 every 8 connection(s)
I20260812 02:37:32.245625 11363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:37:32.251811 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 11363
I20260812 02:37:32.251957 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:37:32.252285 3582 external_mini_cluster.cc:1428] Setting key 18e42707e76e768eacca871762e70344
I20260812 02:37:32.253398 11334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:32.247842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:41017 (local address 127.3.127.190:38147)
0812 02:37:32.248021 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:32.248025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:32.249496 (+ 1471us) server_negotiation.cc:407] Connection header received
0812 02:37:32.251114 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:32.251117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:32.251174 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:32.251261 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:32.252825 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:32.252830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:32.252833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:37:32.252852 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:32.252882 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:37:32.252895 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:32.253001 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:32.253195 (+ 194us) server_negotiation.cc:299] Negotiation successful
0812 02:37:32.253249 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 02:37:32.254747 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:32.248386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38147 (local address 127.3.127.129:41017)
0812 02:37:32.249315 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:32.249368 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:32.250843 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:32.251489 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:32.251500 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:32.251641 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:32.252431 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:37:32.252433 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:32.252498 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:37:32.252607 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:32.252998 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:32.253011 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:32.253018 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 02:37:32.253260 (+ 242us) client_negotiation.cc:240] Negotiation successful
0812 02:37:32.253549 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":170,"threads_started":1}
I20260812 02:37:32.254832 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38147
--builtin_ntp_servers=127.3.127.148:34061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 02:37:32.256462 11568 heartbeater.cc:347] Connected to a master server at 127.3.127.190:38147
I20260812 02:37:32.256814 11568 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:32.257376 11568 heartbeater.cc:511] Master 127.3.127.190:38147 requested a full tablet report, sending...
I20260812 02:37:32.259362 11259 ts_manager.cc:194] Registered new tserver with Master: 0037954a57bf4e0ab82c704b5a229a0e (127.3.127.129:46077)
W20260812 02:37:32.384342 11574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:32.384665 11574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:32.384709 11574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:32.389055 11574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:32.389142 11574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:32.389253 11574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:37:32.394495 11574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:34061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:32.395797 11574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:32.397347 11574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:32.407872 11582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:32.407872 11579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:32.407886 11580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:32.408208 11574 server_base.cc:1034] running on GCE node
I20260812 02:37:32.408795 11574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:32.409598 11574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:32.410808 11574 hybrid_clock.cc:648] HybridClock initialized: now 1786502252410774 us; error 46 us; skew 500 ppm
I20260812 02:37:32.413062 11574 webserver.cc:460] Webserver started at http://127.3.127.130:33681/ using document root <none> and password file <none>
I20260812 02:37:32.413756 11574 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:32.413841 11574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:32.414064 11574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:32.416349 11574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "21341b3ed79e4754ae27adb0bf75761a"
format_stamp: "Formatted at 2026-08-12 02:37:32 on dist-test-slave-rmhg"
server_key: "49012ca18fea0034a8771b416acb9b5c"
server_key_iv: "811282491ec696e1d968dbd25f79624c"
server_key_version: "encryptionkey@0"
I20260812 02:37:32.416935 11574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "21341b3ed79e4754ae27adb0bf75761a"
format_stamp: "Formatted at 2026-08-12 02:37:32 on dist-test-slave-rmhg"
server_key: "49012ca18fea0034a8771b416acb9b5c"
server_key_iv: "811282491ec696e1d968dbd25f79624c"
server_key_version: "encryptionkey@0"
I20260812 02:37:32.421182 11574 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:37:32.423897 11594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:32.425375 11574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:32.425503 11574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "21341b3ed79e4754ae27adb0bf75761a"
format_stamp: "Formatted at 2026-08-12 02:37:32 on dist-test-slave-rmhg"
server_key: "49012ca18fea0034a8771b416acb9b5c"
server_key_iv: "811282491ec696e1d968dbd25f79624c"
server_key_version: "encryptionkey@0"
I20260812 02:37:32.425648 11574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:32.430859 11574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:32.431631 11574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:32.431819 11574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:32.432422 11574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:32.433585 11574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:32.433640 11574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:32.433688 11574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:32.433728 11574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:32.443513 11574 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:46733
I20260812 02:37:32.443542 11778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:46733 every 8 connection(s)
I20260812 02:37:32.444635 11574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:37:32.451800 11334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:32.446828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:39119 (local address 127.3.127.190:38147)
0812 02:37:32.446979 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:32.446983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:32.448274 (+ 1291us) server_negotiation.cc:407] Connection header received
0812 02:37:32.449794 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:32.449797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:32.449844 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:32.449929 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:32.451246 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:32.451249 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:32.451252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:37:32.451267 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:32.451291 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:37:32.451300 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:32.451395 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:32.451592 (+ 197us) server_negotiation.cc:299] Negotiation successful
0812 02:37:32.451644 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 02:37:32.452692 11784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:32.447287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38147 (local address 127.3.127.130:39119)
0812 02:37:32.448097 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:32.448153 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:32.449508 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:32.450174 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:32.450182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:32.450293 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:32.450917 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:37:32.450919 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:32.450968 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:37:32.451054 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:32.451387 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:32.451393 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:32.451397 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 02:37:32.451542 (+ 145us) client_negotiation.cc:240] Negotiation successful
0812 02:37:32.451782 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":127,"threads_started":1}
I20260812 02:37:32.453830 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 11574
I20260812 02:37:32.453933 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:37:32.454267 3582 external_mini_cluster.cc:1428] Setting key 632b068ba5c02a1e825d316b40e1b176
I20260812 02:37:32.454584 11780 heartbeater.cc:347] Connected to a master server at 127.3.127.190:38147
I20260812 02:37:32.454872 11780 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:32.455340 11780 heartbeater.cc:511] Master 127.3.127.190:38147 requested a full tablet report, sending...
I20260812 02:37:32.456385 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38147
--builtin_ntp_servers=127.3.127.148:34061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 02:37:32.456610 11259 ts_manager.cc:194] Registered new tserver with Master: 21341b3ed79e4754ae27adb0bf75761a (127.3.127.130:46733)
W20260812 02:37:32.585721 11785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:32.586037 11785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:32.586079 11785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:32.590798 11785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:32.590899 11785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:32.590969 11785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:37:32.596997 11785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:34061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:32.598315 11785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:32.599889 11785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:32.611231 11799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:32.611238 11793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:32.611588 11797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:32.611379 11785 server_base.cc:1034] running on GCE node
I20260812 02:37:32.612131 11785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:32.612860 11785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:32.614073 11785 hybrid_clock.cc:648] HybridClock initialized: now 1786502252614028 us; error 63 us; skew 500 ppm
I20260812 02:37:32.616406 11785 webserver.cc:460] Webserver started at http://127.3.127.131:37617/ using document root <none> and password file <none>
I20260812 02:37:32.617018 11785 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:32.617084 11785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:32.617311 11785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:32.619681 11785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "d6a46edde11c490787ad1b17f7f1f9f9"
format_stamp: "Formatted at 2026-08-12 02:37:32 on dist-test-slave-rmhg"
server_key: "3470b49e9ed96e64336baa6aa18ff205"
server_key_iv: "e3dd528bff86c1196ecd8baad2e91849"
server_key_version: "encryptionkey@0"
I20260812 02:37:32.620267 11785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "d6a46edde11c490787ad1b17f7f1f9f9"
format_stamp: "Formatted at 2026-08-12 02:37:32 on dist-test-slave-rmhg"
server_key: "3470b49e9ed96e64336baa6aa18ff205"
server_key_iv: "e3dd528bff86c1196ecd8baad2e91849"
server_key_version: "encryptionkey@0"
I20260812 02:37:32.624625 11785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 02:37:32.627591 11813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:32.629302 11785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:32.629448 11785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "d6a46edde11c490787ad1b17f7f1f9f9"
format_stamp: "Formatted at 2026-08-12 02:37:32 on dist-test-slave-rmhg"
server_key: "3470b49e9ed96e64336baa6aa18ff205"
server_key_iv: "e3dd528bff86c1196ecd8baad2e91849"
server_key_version: "encryptionkey@0"
I20260812 02:37:32.629590 11785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:32.635032 11785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:32.635773 11785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:32.635965 11785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:32.636631 11785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:32.637844 11785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:32.637898 11785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:32.637945 11785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:32.637975 11785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:32.647203 11785 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:36515
I20260812 02:37:32.647279 12003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:36515 every 8 connection(s)
I20260812 02:37:32.648244 11785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:37:32.655019 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 11785
I20260812 02:37:32.655175 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:37:32.655457 3582 external_mini_cluster.cc:1428] Setting key 1e5a9eb4b4f3444e194180408ba5d82f
I20260812 02:37:32.655994 11334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:32.650528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:47607 (local address 127.3.127.190:38147)
0812 02:37:32.650735 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:32.650740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:32.652266 (+ 1526us) server_negotiation.cc:407] Connection header received
0812 02:37:32.653771 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:32.653774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:32.653822 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:32.653936 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:32.655503 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:32.655507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:32.655511 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:37:32.655525 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:32.655551 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:37:32.655563 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:32.655664 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:32.655798 (+ 134us) server_negotiation.cc:299] Negotiation successful
0812 02:37:32.655851 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 02:37:32.657191 12011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:32.651058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38147 (local address 127.3.127.131:47607)
0812 02:37:32.652041 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:32.652125 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:32.653500 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:32.654168 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:32.654176 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:32.654281 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:32.655175 (+ 894us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:37:32.655177 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:32.655240 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:37:32.655333 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:32.655643 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:32.655654 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:32.655661 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 02:37:32.655846 (+ 185us) client_negotiation.cc:240] Negotiation successful
0812 02:37:32.656147 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":147,"threads_started":1}
I20260812 02:37:32.658368 12008 heartbeater.cc:347] Connected to a master server at 127.3.127.190:38147
I20260812 02:37:32.658632 12008 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:32.659078 12008 heartbeater.cc:511] Master 127.3.127.190:38147 requested a full tablet report, sending...
I20260812 02:37:32.660177 11259 ts_manager.cc:194] Registered new tserver with Master: d6a46edde11c490787ad1b17f7f1f9f9 (127.3.127.131:36515)
I20260812 02:37:32.670320 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:37:32.674332 11334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:32.672574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57576 (local address 127.3.127.190:38147)
0812 02:37:32.672750 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:32.672754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:32.672897 (+ 143us) server_negotiation.cc:407] Connection header received
0812 02:37:32.673034 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:32.673037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:32.673077 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:32.673180 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:32.673897 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:32.673901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:32.673904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:37:32.673918 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:32.673939 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:37:32.673947 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:32.674023 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:32.674137 (+ 114us) server_negotiation.cc:299] Negotiation successful
0812 02:37:32.674187 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 02:37:32.677687 11259 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:37:32.680999 11259 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:37:32.694777 12028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:32.691158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:46077 (local address 127.0.0.1:40458)
0812 02:37:32.692023 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:32.692078 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:32.692213 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:32.693051 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:32.693058 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:32.693143 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:32.693854 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:37:32.693856 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:32.693879 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:37:32.693968 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:32.694407 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:32.694417 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:32.694425 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 02:37:32.694537 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 02:37:32.694616 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"spinlock_wait_cycles":5376,"thread_start_us":98,"threads_started":1}
I20260812 02:37:32.694937 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:32.691178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:46733 (local address 127.0.0.1:35938)
0812 02:37:32.692238 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:32.692252 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:32.692336 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:32.693140 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:32.693144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:32.693184 (+ 40us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:32.693894 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:37:32.693896 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:32.693908 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:37:32.693972 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:32.694684 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:32.694688 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:32.694709 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 02:37:32.694769 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 02:37:32.694805 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":998,"thread_start_us":103,"threads_started":1}
I20260812 02:37:32.695221 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:32.691987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:36515 (local address 127.0.0.1:54644)
0812 02:37:32.693521 (+ 1534us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:32.693534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:32.693619 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:32.694067 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:32.694070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:32.694085 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:32.694612 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:37:32.694614 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:32.694624 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:37:32.694647 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:32.694997 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:32.695000 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:32.695003 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 02:37:32.695069 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 02:37:32.695115 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1470,"thread_start_us":107,"threads_started":1}
I20260812 02:37:32.695271 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:32.691324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40458 (local address 127.3.127.129:46077)
0812 02:37:32.691673 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:32.691687 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:32.692328 (+ 641us) server_negotiation.cc:407] Connection header received
0812 02:37:32.692399 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:32.692405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:32.692627 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:32.692768 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:32.694131 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:32.694138 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:32.694152 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:37:32.694166 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:32.694214 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:37:32.694256 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:32.694360 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:32.694978 (+ 618us) server_negotiation.cc:299] Negotiation successful
0812 02:37:32.695089 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":114,"threads_started":1}
I20260812 02:37:32.695695 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:32.692033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35938 (local address 127.3.127.130:46733)
0812 02:37:32.692699 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:32.692712 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:32.692735 (+ 23us) server_negotiation.cc:407] Connection header received
0812 02:37:32.692785 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:32.692791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:32.692950 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:32.693462 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:32.694431 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:32.694437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:32.694449 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:37:32.694460 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:32.694499 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:37:32.694563 (+ 64us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:32.695335 (+ 772us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:32.695385 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 02:37:32.695461 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":328,"threads_started":1}
I20260812 02:37:32.695734 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:32.692415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.3.127.131:36515)
0812 02:37:32.692780 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:32.692792 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:32.693640 (+ 848us) server_negotiation.cc:407] Connection header received
0812 02:37:32.693740 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:32.693746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:32.693922 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:32.694071 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:32.694740 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:32.694765 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:32.694778 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:37:32.694789 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:32.694828 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:37:32.694857 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:32.694952 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:32.695475 (+ 523us) server_negotiation.cc:299] Negotiation successful
0812 02:37:32.695539 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":170,"threads_started":1}
I20260812 02:37:32.698351 11680 tablet_service.cc:1467] Processing CreateTablet for tablet 14b956d8e0de438ab8e697500f25872b (DEFAULT_TABLE table=test-table [id=c0637caad743491b82f2ceaa36b3854e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:32.698724 11897 tablet_service.cc:1467] Processing CreateTablet for tablet 14b956d8e0de438ab8e697500f25872b (DEFAULT_TABLE table=test-table [id=c0637caad743491b82f2ceaa36b3854e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:32.698556 11466 tablet_service.cc:1467] Processing CreateTablet for tablet 14b956d8e0de438ab8e697500f25872b (DEFAULT_TABLE table=test-table [id=c0637caad743491b82f2ceaa36b3854e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:32.699506 11680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b956d8e0de438ab8e697500f25872b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:32.699712 11897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b956d8e0de438ab8e697500f25872b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:32.699712 11466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b956d8e0de438ab8e697500f25872b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:32.706899 12034 tablet_bootstrap.cc:492] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a: Bootstrap starting.
I20260812 02:37:32.708340 12035 tablet_bootstrap.cc:492] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9: Bootstrap starting.
I20260812 02:37:32.708657 12036 tablet_bootstrap.cc:492] T 14b956d8e0de438ab8e697500f25872b P 0037954a57bf4e0ab82c704b5a229a0e: Bootstrap starting.
I20260812 02:37:32.709717 12034 tablet_bootstrap.cc:654] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:32.710680 12036 tablet_bootstrap.cc:654] T 14b956d8e0de438ab8e697500f25872b P 0037954a57bf4e0ab82c704b5a229a0e: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:32.710855 12035 tablet_bootstrap.cc:654] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:32.710891 12034 log.cc:824] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:32.711642 12036 log.cc:824] T 14b956d8e0de438ab8e697500f25872b P 0037954a57bf4e0ab82c704b5a229a0e: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:32.711712 12035 log.cc:824] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:32.713948 12034 tablet_bootstrap.cc:492] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a: No bootstrap required, opened a new log
I20260812 02:37:32.713963 12035 tablet_bootstrap.cc:492] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9: No bootstrap required, opened a new log
I20260812 02:37:32.713963 12036 tablet_bootstrap.cc:492] T 14b956d8e0de438ab8e697500f25872b P 0037954a57bf4e0ab82c704b5a229a0e: No bootstrap required, opened a new log
I20260812 02:37:32.714183 12034 ts_tablet_manager.cc:1397] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 02:37:32.714183 12036 ts_tablet_manager.cc:1397] T 14b956d8e0de438ab8e697500f25872b P 0037954a57bf4e0ab82c704b5a229a0e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 02:37:32.714183 12035 ts_tablet_manager.cc:1397] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 02:37:32.718116 12034 raft_consensus.cc:348] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0037954a57bf4e0ab82c704b5a229a0e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46077 } } peers { permanent_uuid: "21341b3ed79e4754ae27adb0bf75761a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 46733 } } peers { permanent_uuid: "d6a46edde11c490787ad1b17f7f1f9f9" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36515 } }
I20260812 02:37:32.718466 12034 raft_consensus.cc:374] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:32.718528 12034 raft_consensus.cc:729] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21341b3ed79e4754ae27adb0bf75761a, State: Initialized, Role: FOLLOWER
I20260812 02:37:32.719008 12034 consensus_queue.cc:260] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0037954a57bf4e0ab82c704b5a229a0e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46077 } } peers { permanent_uuid: "21341b3ed79e4754ae27adb0bf75761a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 46733 } } peers { permanent_uuid: "d6a46edde11c490787ad1b17f7f1f9f9" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36515 } }
I20260812 02:37:32.718909 12035 raft_consensus.cc:348] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0037954a57bf4e0ab82c704b5a229a0e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46077 } } peers { permanent_uuid: "21341b3ed79e4754ae27adb0bf75761a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 46733 } } peers { permanent_uuid: "d6a46edde11c490787ad1b17f7f1f9f9" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36515 } }
I20260812 02:37:32.718909 12036 raft_consensus.cc:348] T 14b956d8e0de438ab8e697500f25872b P 0037954a57bf4e0ab82c704b5a229a0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0037954a57bf4e0ab82c704b5a229a0e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46077 } } peers { permanent_uuid: "21341b3ed79e4754ae27adb0bf75761a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 46733 } } peers { permanent_uuid: "d6a46edde11c490787ad1b17f7f1f9f9" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36515 } }
I20260812 02:37:32.719177 12035 raft_consensus.cc:374] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:32.719177 12036 raft_consensus.cc:374] T 14b956d8e0de438ab8e697500f25872b P 0037954a57bf4e0ab82c704b5a229a0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:32.719239 12036 raft_consensus.cc:729] T 14b956d8e0de438ab8e697500f25872b P 0037954a57bf4e0ab82c704b5a229a0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0037954a57bf4e0ab82c704b5a229a0e, State: Initialized, Role: FOLLOWER
I20260812 02:37:32.719249 12035 raft_consensus.cc:729] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a46edde11c490787ad1b17f7f1f9f9, State: Initialized, Role: FOLLOWER
I20260812 02:37:32.719779 11780 heartbeater.cc:503] Master 127.3.127.190:38147 was elected leader, sending a full tablet report...
I20260812 02:37:32.719718 12036 consensus_queue.cc:260] T 14b956d8e0de438ab8e697500f25872b P 0037954a57bf4e0ab82c704b5a229a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0037954a57bf4e0ab82c704b5a229a0e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46077 } } peers { permanent_uuid: "21341b3ed79e4754ae27adb0bf75761a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 46733 } } peers { permanent_uuid: "d6a46edde11c490787ad1b17f7f1f9f9" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36515 } }
I20260812 02:37:32.720103 12034 ts_tablet_manager.cc:1428] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 02:37:32.720347 12035 consensus_queue.cc:260] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0037954a57bf4e0ab82c704b5a229a0e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46077 } } peers { permanent_uuid: "21341b3ed79e4754ae27adb0bf75761a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 46733 } } peers { permanent_uuid: "d6a46edde11c490787ad1b17f7f1f9f9" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36515 } }
I20260812 02:37:32.720635 11568 heartbeater.cc:503] Master 127.3.127.190:38147 was elected leader, sending a full tablet report...
I20260812 02:37:32.720774 12036 ts_tablet_manager.cc:1428] T 14b956d8e0de438ab8e697500f25872b P 0037954a57bf4e0ab82c704b5a229a0e: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260812 02:37:32.721321 12035 ts_tablet_manager.cc:1428] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 02:37:32.721691 12008 heartbeater.cc:503] Master 127.3.127.190:38147 was elected leader, sending a full tablet report...
W20260812 02:37:32.746990 11571 tablet.cc:2369] T 14b956d8e0de438ab8e697500f25872b P 0037954a57bf4e0ab82c704b5a229a0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:37:32.890925 12044 raft_consensus.cc:482] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:37:32.891182 12044 raft_consensus.cc:504] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0037954a57bf4e0ab82c704b5a229a0e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46077 } } peers { permanent_uuid: "21341b3ed79e4754ae27adb0bf75761a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 46733 } } peers { permanent_uuid: "d6a46edde11c490787ad1b17f7f1f9f9" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36515 } }
I20260812 02:37:32.892383 12044 leader_election.cc:288] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0037954a57bf4e0ab82c704b5a229a0e (127.3.127.129:46077), d6a46edde11c490787ad1b17f7f1f9f9 (127.3.127.131:36515)
I20260812 02:37:32.894472 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:32.892853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:54651 (local address 127.3.127.129:46077)
0812 02:37:32.893024 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:32.893030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:32.893049 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:37:32.893103 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:32.893106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:32.893173 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:32.893267 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:32.894064 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:32.894093 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:32.894096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:37:32.894111 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:32.894137 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:37:32.894148 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:32.894241 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:32.894300 (+ 59us) server_negotiation.cc:299] Negotiation successful
0812 02:37:32.894338 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 02:37:32.894472 11784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:32.892698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:46077 (local address 127.3.127.130:54651)
0812 02:37:32.892850 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:32.892866 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:32.892975 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:32.893340 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:32.893344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:32.893360 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:32.893902 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:37:32.893904 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:32.893917 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:37:32.893947 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:32.894222 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:32.894225 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:32.894229 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 02:37:32.894301 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 02:37:32.894338 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 02:37:32.895117 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:32.893004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:36515 (local address 127.3.127.130:48255)
0812 02:37:32.893562 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:32.893602 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:32.893703 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:32.893955 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:32.893959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:32.893973 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:32.894453 (+ 480us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:37:32.894455 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:32.894467 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:37:32.894495 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:32.894838 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:32.894841 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:32.894843 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 02:37:32.894938 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 02:37:32.894973 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":202,"threads_started":1}
I20260812 02:37:32.895162 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:32.893207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:48255 (local address 127.3.127.131:36515)
0812 02:37:32.893327 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:32.893331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:32.893722 (+ 391us) server_negotiation.cc:407] Connection header received
0812 02:37:32.893780 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:32.893783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:32.893826 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:32.893938 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:32.894654 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:32.894667 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:32.894670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:37:32.894684 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:32.894705 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:37:32.894715 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:32.894775 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:32.894975 (+ 200us) server_negotiation.cc:299] Negotiation successful
0812 02:37:32.895026 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 02:37:32.895460 11496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b956d8e0de438ab8e697500f25872b" candidate_uuid: "21341b3ed79e4754ae27adb0bf75761a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0037954a57bf4e0ab82c704b5a229a0e" is_pre_election: true
I20260812 02:37:32.895830 11496 raft_consensus.cc:2454] T 14b956d8e0de438ab8e697500f25872b P 0037954a57bf4e0ab82c704b5a229a0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21341b3ed79e4754ae27adb0bf75761a in term 0.
I20260812 02:37:32.895951 11931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b956d8e0de438ab8e697500f25872b" candidate_uuid: "21341b3ed79e4754ae27adb0bf75761a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a46edde11c490787ad1b17f7f1f9f9" is_pre_election: true
I20260812 02:37:32.896252 11931 raft_consensus.cc:2454] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21341b3ed79e4754ae27adb0bf75761a in term 0.
I20260812 02:37:32.896377 11601 leader_election.cc:302] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0037954a57bf4e0ab82c704b5a229a0e, 21341b3ed79e4754ae27adb0bf75761a; no voters:
I20260812 02:37:32.896701 12044 raft_consensus.cc:2781] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:37:32.896792 12044 raft_consensus.cc:482] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:37:32.896839 12044 raft_consensus.cc:3037] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a [term 0 FOLLOWER]: Advancing to term 1
W20260812 02:37:32.899296 12010 tablet.cc:2369] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:37:32.946581 11782 tablet.cc:2369] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:37:32.980455 12044 raft_consensus.cc:504] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0037954a57bf4e0ab82c704b5a229a0e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46077 } } peers { permanent_uuid: "21341b3ed79e4754ae27adb0bf75761a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 46733 } } peers { permanent_uuid: "d6a46edde11c490787ad1b17f7f1f9f9" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36515 } }
I20260812 02:37:32.981057 12044 leader_election.cc:288] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a [CANDIDATE]: Term 1 election: Requested vote from peers 0037954a57bf4e0ab82c704b5a229a0e (127.3.127.129:46077), d6a46edde11c490787ad1b17f7f1f9f9 (127.3.127.131:36515)
I20260812 02:37:32.981673 11931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b956d8e0de438ab8e697500f25872b" candidate_uuid: "21341b3ed79e4754ae27adb0bf75761a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a46edde11c490787ad1b17f7f1f9f9"
I20260812 02:37:32.981673 11496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b956d8e0de438ab8e697500f25872b" candidate_uuid: "21341b3ed79e4754ae27adb0bf75761a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0037954a57bf4e0ab82c704b5a229a0e"
I20260812 02:37:32.981846 11931 raft_consensus.cc:3037] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:32.981846 11496 raft_consensus.cc:3037] T 14b956d8e0de438ab8e697500f25872b P 0037954a57bf4e0ab82c704b5a229a0e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:32.983328 11931 raft_consensus.cc:2454] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21341b3ed79e4754ae27adb0bf75761a in term 1.
I20260812 02:37:32.983330 11496 raft_consensus.cc:2454] T 14b956d8e0de438ab8e697500f25872b P 0037954a57bf4e0ab82c704b5a229a0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21341b3ed79e4754ae27adb0bf75761a in term 1.
I20260812 02:37:32.983781 11601 leader_election.cc:302] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21341b3ed79e4754ae27adb0bf75761a, d6a46edde11c490787ad1b17f7f1f9f9; no voters:
I20260812 02:37:32.984030 12044 raft_consensus.cc:2781] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:32.984285 12044 raft_consensus.cc:686] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a [term 1 LEADER]: Becoming Leader. State: Replica: 21341b3ed79e4754ae27adb0bf75761a, State: Running, Role: LEADER
I20260812 02:37:32.984752 12044 consensus_queue.cc:237] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0037954a57bf4e0ab82c704b5a229a0e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46077 } } peers { permanent_uuid: "21341b3ed79e4754ae27adb0bf75761a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 46733 } } peers { permanent_uuid: "d6a46edde11c490787ad1b17f7f1f9f9" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36515 } }
I20260812 02:37:32.988754 11256 catalog_manager.cc:5495] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a reported cstate change: term changed from 0 to 1, leader changed from <none> to 21341b3ed79e4754ae27adb0bf75761a (127.3.127.130). New cstate: current_term: 1 leader_uuid: "21341b3ed79e4754ae27adb0bf75761a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0037954a57bf4e0ab82c704b5a229a0e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21341b3ed79e4754ae27adb0bf75761a" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 46733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6a46edde11c490787ad1b17f7f1f9f9" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36515 } health_report { overall_health: UNKNOWN } } }
I20260812 02:37:33.067426 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:33.066063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35950 (local address 127.3.127.130:46733)
0812 02:37:33.066196 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:33.066200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:33.066238 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:37:33.066328 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:33.066330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:33.066397 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:33.066458 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:33.067111 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:33.067114 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:33.067116 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:37:33.067128 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:33.067148 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:37:33.067157 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:33.067230 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:33.067276 (+ 46us) server_negotiation.cc:299] Negotiation successful
0812 02:37:33.067311 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 02:37:33.074330 11931 raft_consensus.cc:1261] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9 [term 1 FOLLOWER]: Refusing update from remote peer 21341b3ed79e4754ae27adb0bf75761a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:33.074330 11496 raft_consensus.cc:1261] T 14b956d8e0de438ab8e697500f25872b P 0037954a57bf4e0ab82c704b5a229a0e [term 1 FOLLOWER]: Refusing update from remote peer 21341b3ed79e4754ae27adb0bf75761a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:33.075207 12044 consensus_queue.cc:1035] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a46edde11c490787ad1b17f7f1f9f9" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:33.075470 12052 consensus_queue.cc:1035] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0037954a57bf4e0ab82c704b5a229a0e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 46077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:33.084828 12057 mvcc.cc:205] Tried to move back new op lower bound from 7317513228583174144 to 7317513228228771840. Current Snapshot: MvccSnapshot[applied={T|T < 7317513228583174144}]
I20260812 02:37:33.093134 11256 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:33.093381 11256 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:33.096105 11256 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84c34cc6d9254609bc7d073d9247e7dc: Sending DeleteTablet for 3 replicas of tablet 14b956d8e0de438ab8e697500f25872b
I20260812 02:37:33.097222 11680 tablet_service.cc:1514] Processing DeleteTablet for tablet 14b956d8e0de438ab8e697500f25872b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:33 UTC) from {username='slave'} at 127.0.0.1:35938
I20260812 02:37:33.097324 11897 tablet_service.cc:1514] Processing DeleteTablet for tablet 14b956d8e0de438ab8e697500f25872b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:33 UTC) from {username='slave'} at 127.0.0.1:54644
I20260812 02:37:33.097862 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 11363
I20260812 02:37:33.098074 12072 tablet_replica.cc:330] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9: stopping tablet replica
I20260812 02:37:33.110387 12072 raft_consensus.cc:2229] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:33.111037 12072 raft_consensus.cc:2258] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:33.114542 12072 ts_tablet_manager.cc:1905] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:33.115078 12071 tablet_replica.cc:330] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a: stopping tablet replica
I20260812 02:37:33.115470 12071 raft_consensus.cc:2229] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:37:33.115980 12071 raft_consensus.cc:2258] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:33.117854 12071 ts_tablet_manager.cc:1905] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:33.118090 12072 ts_tablet_manager.cc:1918] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:37:33.118248 12072 log.cc:1196] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/14b956d8e0de438ab8e697500f25872b
I20260812 02:37:33.118644 12072 ts_tablet_manager.cc:1939] T 14b956d8e0de438ab8e697500f25872b P d6a46edde11c490787ad1b17f7f1f9f9: Deleting consensus metadata
I20260812 02:37:33.120669 12071 ts_tablet_manager.cc:1918] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:37:33.120788 12071 log.cc:1196] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/14b956d8e0de438ab8e697500f25872b
I20260812 02:37:33.121095 12071 ts_tablet_manager.cc:1939] T 14b956d8e0de438ab8e697500f25872b P 21341b3ed79e4754ae27adb0bf75761a: Deleting consensus metadata
I20260812 02:37:33.121481 11240 catalog_manager.cc:4841] TS d6a46edde11c490787ad1b17f7f1f9f9 (127.3.127.131:36515): tablet 14b956d8e0de438ab8e697500f25872b (table test-table [id=c0637caad743491b82f2ceaa36b3854e]) successfully deleted
I20260812 02:37:33.122414 11237 catalog_manager.cc:4841] TS 21341b3ed79e4754ae27adb0bf75761a (127.3.127.130:46733): tablet 14b956d8e0de438ab8e697500f25872b (table test-table [id=c0637caad743491b82f2ceaa36b3854e]) successfully deleted
W20260812 02:37:33.123463 11240 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.127.129:46077 (error 108)
I20260812 02:37:33.124328 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 11574
I20260812 02:37:33.124619 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:33.124276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:46077 (local address 127.0.0.1:40460)
0812 02:37:33.124428 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:33.124518 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.129:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260812 02:37:33.129513 11240 catalog_manager.cc:4584] TS 0037954a57bf4e0ab82c704b5a229a0e (127.3.127.129:46077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14b956d8e0de438ab8e697500f25872b: Network error: Client connection negotiation failed: client connection to 127.3.127.129:46077: connect: Connection refused (error 111)
I20260812 02:37:33.139321 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 11785
I20260812 02:37:33.156384 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 11214
2026-08-12T02:37:33Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1448 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260812 02:37:33.185675 3582 mini_oidc.cc:152] Starting JWKS server
I20260812 02:37:33.187616 3582 webserver.cc:460] Webserver started at https://127.0.0.1:45325/ using document root <none> and password file <none>
I20260812 02:37:33.187820 3582 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 02:37:33.188131 3582 webserver.cc:460] Webserver started at http://127.0.0.1:37257/ using document root <none> and password file <none>
2026-08-12T02:37:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:37:33Z Disabled control of system clock
I20260812 02:37:33.198799 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:40551
--webserver_interface=127.3.127.190
--webserver_port=0
--builtin_ntp_servers=127.3.127.148:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:40551
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45325/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786502224378336-3582-0/testchainca.cert with env {}
W20260812 02:37:33.318663 12081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:33.318996 12081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:33.319038 12081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:33.323405 12081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:33.323498 12081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:33.323518 12081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:33.323535 12081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:33.323549 12081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 02:37:33.323565 12081 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 02:37:33.323576 12081 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45325/jwks.json
I20260812 02:37:33.329618 12081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:40551
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:40551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45325/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786502224378336-3582-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:33.330798 12081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:33.331825 12081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:33.341282 12089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:33.341282 12095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:33.341499 12081 server_base.cc:1034] running on GCE node
W20260812 02:37:33.341284 12091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:33.342377 12081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:33.343623 12081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:33.344821 12081 hybrid_clock.cc:648] HybridClock initialized: now 1786502253344804 us; error 30 us; skew 500 ppm
I20260812 02:37:33.347194 12081 webserver.cc:460] Webserver started at http://127.3.127.190:38359/ using document root <none> and password file <none>
I20260812 02:37:33.347815 12081 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:33.347879 12081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:33.348080 12081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:33.350353 12081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "5ae564dc72db47928a3ce9022fb3a554"
format_stamp: "Formatted at 2026-08-12 02:37:33 on dist-test-slave-rmhg"
server_key: "c45c424c3a0cf5320d380e5fa94a0b17"
server_key_iv: "6aa90bf547fa33ba23c298e76a467e32"
server_key_version: "encryptionkey@0"
I20260812 02:37:33.350920 12081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "5ae564dc72db47928a3ce9022fb3a554"
format_stamp: "Formatted at 2026-08-12 02:37:33 on dist-test-slave-rmhg"
server_key: "c45c424c3a0cf5320d380e5fa94a0b17"
server_key_iv: "6aa90bf547fa33ba23c298e76a467e32"
server_key_version: "encryptionkey@0"
I20260812 02:37:33.354940 12081 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 02:37:33.357721 12105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:33.359163 12081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:33.359306 12081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "5ae564dc72db47928a3ce9022fb3a554"
format_stamp: "Formatted at 2026-08-12 02:37:33 on dist-test-slave-rmhg"
server_key: "c45c424c3a0cf5320d380e5fa94a0b17"
server_key_iv: "6aa90bf547fa33ba23c298e76a467e32"
server_key_version: "encryptionkey@0"
I20260812 02:37:33.359429 12081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:33.386251 12081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:33.387046 12081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:33.387223 12081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:33.394649 12081 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:40551
I20260812 02:37:33.394659 12183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:40551 every 8 connection(s)
I20260812 02:37:33.395673 12081 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:37:33.396384 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 12081
I20260812 02:37:33.396494 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:37:33.396762 3582 external_mini_cluster.cc:1428] Setting key ee7668661026df18271224758360213d
I20260812 02:37:33.399482 12184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:33.405484 12184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554: Bootstrap starting.
I20260812 02:37:33.406790 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:33.398295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40894 (local address 127.3.127.190:40551)
0812 02:37:33.399000 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:33.399017 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:33.399056 (+ 39us) server_negotiation.cc:407] Connection header received
0812 02:37:33.399811 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:33.399832 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:33.400256 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:33.400667 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:33.401135 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:33.402198 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:33.403138 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:33.403656 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:33.403814 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:33.404780 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:33.404785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:33.404799 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:37:33.405270 (+ 471us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:33.405333 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:37:33.405357 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:33.405450 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:33.405595 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 02:37:33.405907 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":150,"threads_started":1}
I20260812 02:37:33.407919 12184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:33.408869 12184 log.cc:824] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:33.410213 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:37:33.411137 12184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554: No bootstrap required, opened a new log
I20260812 02:37:33.415151 12184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae564dc72db47928a3ce9022fb3a554" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40551 } }
I20260812 02:37:33.415411 12184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:33.415484 12184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae564dc72db47928a3ce9022fb3a554, State: Initialized, Role: FOLLOWER
I20260812 02:37:33.415949 12184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae564dc72db47928a3ce9022fb3a554" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40551 } }
I20260812 02:37:33.416095 12184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:37:33.416167 12184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:37:33.416271 12184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:33.417332 12184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae564dc72db47928a3ce9022fb3a554" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40551 } }
I20260812 02:37:33.417695 12184 leader_election.cc:302] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ae564dc72db47928a3ce9022fb3a554; no voters:
I20260812 02:37:33.418041 12184 leader_election.cc:288] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:37:33.418183 12189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:33.418391 12189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554 [term 1 LEADER]: Becoming Leader. State: Replica: 5ae564dc72db47928a3ce9022fb3a554, State: Running, Role: LEADER
I20260812 02:37:33.418737 12189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae564dc72db47928a3ce9022fb3a554" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40551 } }
I20260812 02:37:33.419350 12184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:37:33.420395 12195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ae564dc72db47928a3ce9022fb3a554. Latest consensus state: current_term: 1 leader_uuid: "5ae564dc72db47928a3ce9022fb3a554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae564dc72db47928a3ce9022fb3a554" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40551 } } }
I20260812 02:37:33.420557 12195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:33.420954 12206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:37:33.421609 12190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ae564dc72db47928a3ce9022fb3a554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae564dc72db47928a3ce9022fb3a554" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40551 } } }
I20260812 02:37:33.421703 12190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:33.423981 12206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:37:33.430032 12206 catalog_manager.cc:1347] Generated new cluster ID: 3b36ebaf881844fea847e7bf87e2453d
I20260812 02:37:33.430124 12206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:37:33.443900 12206 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:37:33.445035 12206 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:37:33.471858 12206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ae564dc72db47928a3ce9022fb3a554: Generated new TSK 0
I20260812 02:37:33.472702 12206 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 02:37:33.512744 12234 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260812 02:37:33.512991 12234 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260812 02:37:33.514796 12232 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:33.488352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:40551 (local address 127.0.0.1:40896)
0812 02:37:33.488771 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:33.488788 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:33.488948 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:33.489437 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:33.489440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:33.489451 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0812 02:37:33.489601 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:33.489608 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:33.490399 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:33.490403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:33.491399 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:33.491411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:33.491847 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:33.491852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:33.491949 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:33.492013 (+ 64us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 02:37:33.513231 (+ 21218us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45325/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 02:37:33.513487 (+ 256us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.127.190:40551: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45325/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":115,"threads_started":1}
I20260812 02:37:33.515442 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:33.488464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40896 (local address 127.3.127.190:40551)
0812 02:37:33.488628 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:33.488633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:33.488872 (+ 239us) server_negotiation.cc:407] Connection header received
0812 02:37:33.489211 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:33.489215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:33.489278 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:33.489381 (+ 103us) server_negotiation.cc:226] Negotiated authn=JWT
0812 02:37:33.489736 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:33.490276 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:33.491517 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:33.491741 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:33.491821 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:33.492262 (+ 441us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 02:37:33.512853 (+ 20591us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45325/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 02:37:33.515298 (+ 2445us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40896: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45325/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260812 02:37:33.515517 12187 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40896: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45325/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.127.190:40551: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45325/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260812 02:37:33.515684 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 12081
2026-08-12T02:37:33Z chronyd exiting
I20260812 02:37:33.569253 3582 test_util.cc:175] -----------------------------------------------
I20260812 02:37:33.569350 3582 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786502224378336-3582-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (385 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 02:37:33.570655 3582 mini_oidc.cc:152] Starting JWKS server
I20260812 02:37:33.571564 3582 webserver.cc:460] Webserver started at https://127.0.0.1:42139/ using document root <none> and password file <none>
I20260812 02:37:33.571673 3582 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 02:37:33.571938 3582 webserver.cc:460] Webserver started at http://127.0.0.1:45141/ using document root <none> and password file <none>
2026-08-12T02:37:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:37:33Z Disabled control of system clock
I20260812 02:37:33.583016 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:39665
--webserver_interface=127.3.127.190
--webserver_port=0
--builtin_ntp_servers=127.3.127.148:40543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:39665
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42139/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786502224378336-3582-0/testchainca.cert with env {}
W20260812 02:37:33.700026 12244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:33.700343 12244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:33.700402 12244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:33.704545 12244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:33.704632 12244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:33.704653 12244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:33.704675 12244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:33.704694 12244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 02:37:33.704717 12244 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 02:37:33.704743 12244 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42139/jwks.json
I20260812 02:37:33.710489 12244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:40543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:39665
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:39665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42139/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786502224378336-3582-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:33.711679 12244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:33.712733 12244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:33.721764 12254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:33.721827 12253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:33.721803 12257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:33.722000 12244 server_base.cc:1034] running on GCE node
I20260812 02:37:33.722741 12244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:33.723892 12244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:33.725075 12244 hybrid_clock.cc:648] HybridClock initialized: now 1786502253725057 us; error 35 us; skew 500 ppm
I20260812 02:37:33.727092 12244 webserver.cc:460] Webserver started at http://127.3.127.190:44973/ using document root <none> and password file <none>
I20260812 02:37:33.727638 12244 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:33.727707 12244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:33.727878 12244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:33.730013 12244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "ffc36b537e084261a6178bcde260699f"
format_stamp: "Formatted at 2026-08-12 02:37:33 on dist-test-slave-rmhg"
server_key: "c12fb515368cc317dd9a6505c3355450"
server_key_iv: "14806afa6faebba92057526cdccb27ac"
server_key_version: "encryptionkey@0"
I20260812 02:37:33.730567 12244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "ffc36b537e084261a6178bcde260699f"
format_stamp: "Formatted at 2026-08-12 02:37:33 on dist-test-slave-rmhg"
server_key: "c12fb515368cc317dd9a6505c3355450"
server_key_iv: "14806afa6faebba92057526cdccb27ac"
server_key_version: "encryptionkey@0"
I20260812 02:37:33.734540 12244 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:37:33.737188 12268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:33.738559 12244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:33.738708 12244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "ffc36b537e084261a6178bcde260699f"
format_stamp: "Formatted at 2026-08-12 02:37:33 on dist-test-slave-rmhg"
server_key: "c12fb515368cc317dd9a6505c3355450"
server_key_iv: "14806afa6faebba92057526cdccb27ac"
server_key_version: "encryptionkey@0"
I20260812 02:37:33.738819 12244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:33.751446 12244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:33.752214 12244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:33.752396 12244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:33.759455 12244 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:39665
I20260812 02:37:33.759480 12336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:39665 every 8 connection(s)
I20260812 02:37:33.760457 12244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:37:33.763826 12337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:33.769990 12337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f: Bootstrap starting.
I20260812 02:37:33.770421 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 12244
I20260812 02:37:33.770537 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:37:33.770823 3582 external_mini_cluster.cc:1428] Setting key eb059f3f1ca6e93df7b04f2fe91f7e7a
I20260812 02:37:33.772809 12337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:33.773625 12337 log.cc:824] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:33.775871 12337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f: No bootstrap required, opened a new log
I20260812 02:37:33.780289 12337 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc36b537e084261a6178bcde260699f" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 39665 } }
I20260812 02:37:33.780548 12337 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:33.780596 12337 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc36b537e084261a6178bcde260699f, State: Initialized, Role: FOLLOWER
I20260812 02:37:33.780817 12341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:33.772285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44956 (local address 127.3.127.190:39665)
0812 02:37:33.773013 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:33.773030 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:33.773070 (+ 40us) server_negotiation.cc:407] Connection header received
0812 02:37:33.773833 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:33.773855 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:33.774300 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:33.774599 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:33.774974 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:33.776028 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:33.777289 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:33.777825 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:33.777963 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:33.778835 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:33.778841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:33.778855 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:37:33.779269 (+ 414us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:33.779333 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:37:33.779356 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:33.779435 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:33.779590 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 02:37:33.779935 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":140,"threads_started":1}
I20260812 02:37:33.781102 12337 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc36b537e084261a6178bcde260699f" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 39665 } }
I20260812 02:37:33.781292 12337 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:37:33.781364 12337 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:37:33.781462 12337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:33.782655 12337 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc36b537e084261a6178bcde260699f" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 39665 } }
I20260812 02:37:33.783025 12337 leader_election.cc:302] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffc36b537e084261a6178bcde260699f; no voters:
I20260812 02:37:33.783432 12337 leader_election.cc:288] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:37:33.783545 12345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:33.783783 12345 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f [term 1 LEADER]: Becoming Leader. State: Replica: ffc36b537e084261a6178bcde260699f, State: Running, Role: LEADER
I20260812 02:37:33.784147 12345 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc36b537e084261a6178bcde260699f" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 39665 } }
I20260812 02:37:33.784230 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:37:33.784917 12337 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:37:33.786054 12347 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffc36b537e084261a6178bcde260699f. Latest consensus state: current_term: 1 leader_uuid: "ffc36b537e084261a6178bcde260699f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc36b537e084261a6178bcde260699f" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 39665 } } }
I20260812 02:37:33.786365 12346 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffc36b537e084261a6178bcde260699f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc36b537e084261a6178bcde260699f" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 39665 } } }
I20260812 02:37:33.786496 12346 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:33.786563 12347 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:33.787294 12365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:37:33.790155 12365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:37:33.796051 12365 catalog_manager.cc:1347] Generated new cluster ID: 0f0b075648494abc910fd1cb0704c0f2
I20260812 02:37:33.796154 12365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:37:33.821416 12365 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:37:33.822808 12365 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:37:33.834750 12365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffc36b537e084261a6178bcde260699f: Generated new TSK 0
I20260812 02:37:33.835568 12365 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 02:37:33.865031 12390 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 02:37:33.865144 12390 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 02:37:33.865867 12341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:33.849634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.3.127.190:39665)
0812 02:37:33.849793 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:33.849798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:33.850003 (+ 205us) server_negotiation.cc:407] Connection header received
0812 02:37:33.850209 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:33.850211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:33.850268 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:33.850369 (+ 101us) server_negotiation.cc:226] Negotiated authn=JWT
0812 02:37:33.850652 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:33.851271 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:33.852547 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:33.852788 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:33.852874 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:33.853304 (+ 430us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 02:37:33.865489 (+ 12185us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 02:37:33.865747 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44964: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260812 02:37:33.865936 12341 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44964: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260812 02:37:33.866175 12389 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:33.849536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:39665 (local address 127.0.0.1:44964)
0812 02:37:33.849906 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:33.849922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:33.850052 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:33.850378 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:33.850381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:33.850389 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0812 02:37:33.850514 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:33.850520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:33.851385 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:33.851388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:33.852406 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:33.852419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:33.852906 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:33.852910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:33.853019 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:33.853055 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 02:37:33.865728 (+ 12673us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 02:37:33.865953 (+ 225us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.127.190:39665: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":155,"threads_started":1}
I20260812 02:37:33.866721 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 12244
2026-08-12T02:37:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (345 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 02:37:33.915553 3582 mini_oidc.cc:152] Starting JWKS server
I20260812 02:37:33.916404 3582 webserver.cc:460] Webserver started at https://127.0.0.1:43913/ using document root <none> and password file <none>
I20260812 02:37:33.916491 3582 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 02:37:33.916706 3582 webserver.cc:460] Webserver started at http://127.0.0.1:35219/ using document root <none> and password file <none>
2026-08-12T02:37:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:37:33Z Disabled control of system clock
I20260812 02:37:33.927644 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:43423
--webserver_interface=127.3.127.190
--webserver_port=0
--builtin_ntp_servers=127.3.127.148:46331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:43423
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43913/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 02:37:34.044431 12400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:34.044739 12400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:34.044778 12400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:34.048835 12400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:34.048915 12400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:34.048928 12400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:34.048943 12400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:34.048959 12400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 02:37:34.048974 12400 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 02:37:34.048985 12400 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43913/jwks.json
I20260812 02:37:34.053975 12400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:46331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:43423
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:43423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43913/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:34.055114 12400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:34.056524 12400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:34.065315 12410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:34.065315 12409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:34.065482 12400 server_base.cc:1034] running on GCE node
W20260812 02:37:34.065335 12415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:34.066392 12400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:34.067538 12400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:34.068735 12400 hybrid_clock.cc:648] HybridClock initialized: now 1786502254068716 us; error 36 us; skew 500 ppm
I20260812 02:37:34.070905 12400 webserver.cc:460] Webserver started at http://127.3.127.190:46487/ using document root <none> and password file <none>
I20260812 02:37:34.071532 12400 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:34.071609 12400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:34.071831 12400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:34.074122 12400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "925a260c8b764e859214f3e73fcdc122"
format_stamp: "Formatted at 2026-08-12 02:37:34 on dist-test-slave-rmhg"
server_key: "71075d135b6ce4705fc57303bb3e4a75"
server_key_iv: "09ce4ae6c4c9a4ddcde0bbd193f8cd0d"
server_key_version: "encryptionkey@0"
I20260812 02:37:34.074703 12400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "925a260c8b764e859214f3e73fcdc122"
format_stamp: "Formatted at 2026-08-12 02:37:34 on dist-test-slave-rmhg"
server_key: "71075d135b6ce4705fc57303bb3e4a75"
server_key_iv: "09ce4ae6c4c9a4ddcde0bbd193f8cd0d"
server_key_version: "encryptionkey@0"
I20260812 02:37:34.078720 12400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:37:34.081454 12427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:34.082868 12400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 02:37:34.083009 12400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "925a260c8b764e859214f3e73fcdc122"
format_stamp: "Formatted at 2026-08-12 02:37:34 on dist-test-slave-rmhg"
server_key: "71075d135b6ce4705fc57303bb3e4a75"
server_key_iv: "09ce4ae6c4c9a4ddcde0bbd193f8cd0d"
server_key_version: "encryptionkey@0"
I20260812 02:37:34.083132 12400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:34.094604 12400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:34.095360 12400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:34.095549 12400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:34.102758 12400 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:43423
I20260812 02:37:34.102777 12513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:43423 every 8 connection(s)
I20260812 02:37:34.103766 12400 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:37:34.104130 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 12400
I20260812 02:37:34.104260 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:37:34.104578 3582 external_mini_cluster.cc:1428] Setting key 5b2d77397146ce5a75ef59299114605f
I20260812 02:37:34.107463 12514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:34.114671 12514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122: Bootstrap starting.
I20260812 02:37:34.115521 12519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:34.106063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54996 (local address 127.3.127.190:43423)
0812 02:37:34.106789 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:34.106810 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:34.106857 (+ 47us) server_negotiation.cc:407] Connection header received
0812 02:37:34.107747 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:34.107773 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:34.108325 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:34.108719 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:34.109234 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:34.110727 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:34.111685 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:34.112378 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:34.112545 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:34.113365 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:34.113371 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:34.113389 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:37:34.113902 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:34.113980 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:37:34.114006 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:34.114106 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:34.114283 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 02:37:34.114565 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":137,"threads_started":1}
I20260812 02:37:34.117292 12514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:34.118176 12514 log.cc:824] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:34.118515 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:37:34.120517 12514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122: No bootstrap required, opened a new log
I20260812 02:37:34.124243 12514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925a260c8b764e859214f3e73fcdc122" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 43423 } }
I20260812 02:37:34.124524 12514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:34.124562 12514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 925a260c8b764e859214f3e73fcdc122, State: Initialized, Role: FOLLOWER
I20260812 02:37:34.125001 12514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925a260c8b764e859214f3e73fcdc122" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 43423 } }
I20260812 02:37:34.125137 12514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:37:34.125226 12514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:37:34.125313 12514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:34.126427 12514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925a260c8b764e859214f3e73fcdc122" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 43423 } }
I20260812 02:37:34.126778 12514 leader_election.cc:302] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 925a260c8b764e859214f3e73fcdc122; no voters:
I20260812 02:37:34.127080 12514 leader_election.cc:288] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:37:34.127256 12524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:34.127549 12524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122 [term 1 LEADER]: Becoming Leader. State: Replica: 925a260c8b764e859214f3e73fcdc122, State: Running, Role: LEADER
I20260812 02:37:34.127871 12524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925a260c8b764e859214f3e73fcdc122" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 43423 } }
I20260812 02:37:34.128373 12514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:37:34.129557 12525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "925a260c8b764e859214f3e73fcdc122" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925a260c8b764e859214f3e73fcdc122" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 43423 } } }
I20260812 02:37:34.129698 12525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:34.129987 12526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 925a260c8b764e859214f3e73fcdc122. Latest consensus state: current_term: 1 leader_uuid: "925a260c8b764e859214f3e73fcdc122" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925a260c8b764e859214f3e73fcdc122" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 43423 } } }
I20260812 02:37:34.130097 12526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:34.130193 12537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:37:34.133422 12537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:37:34.139084 12537 catalog_manager.cc:1347] Generated new cluster ID: 9b62ef73659248c9b31e32e8967a2274
I20260812 02:37:34.139178 12537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:37:34.146857 12537 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:37:34.148106 12537 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:37:34.154028 12537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 925a260c8b764e859214f3e73fcdc122: Generated new TSK 0
I20260812 02:37:34.154904 12537 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:37:34.174638 12519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:34.169783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55006 (local address 127.3.127.190:43423)
0812 02:37:34.169927 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:34.169932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:34.170096 (+ 164us) server_negotiation.cc:407] Connection header received
0812 02:37:34.170267 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:34.170269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:34.170318 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:34.170408 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0812 02:37:34.170655 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:34.171190 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:34.172402 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:34.172620 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:34.172693 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:34.173001 (+ 308us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 02:37:34.174266 (+ 1265us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43913/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 02:37:34.174485 (+ 219us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55006: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43913/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":54}
W20260812 02:37:34.174710 12519 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55006: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43913/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260812 02:37:34.177507 12562 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:34.169701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:43423 (local address 127.0.0.1:55006)
0812 02:37:34.170009 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:34.170025 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:34.170145 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:34.170425 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:34.170428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:34.170436 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0812 02:37:34.170563 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:34.170569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:34.171292 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:34.171296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:34.172287 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:34.172297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:34.172732 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:34.172736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:34.172825 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:34.172855 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 02:37:34.174429 (+ 1574us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43913/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 02:37:34.177362 (+ 2933us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.127.190:43423: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43913/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":2753,"thread_start_us":89,"threads_started":1}
I20260812 02:37:34.178067 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 12400
E20260812 02:37:34.198014 12563 webserver.cc:533] Webserver: sslize failed:
E20260812 02:37:34.198103 12563 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-12T02:37:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (314 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T02:37:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:37:34Z Disabled control of system clock
I20260812 02:37:34.240558 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:34059
--webserver_interface=127.3.127.190
--webserver_port=0
--builtin_ntp_servers=127.3.127.148:41231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:34059
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786502224378336-3582-0/testchainca.cert with env {}
W20260812 02:37:34.359476 12569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:34.359876 12569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:34.359951 12569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:34.364084 12569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:34.364158 12569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:34.364181 12569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:34.364202 12569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:34.364223 12569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 02:37:34.364248 12569 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 02:37:34.364275 12569 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 02:37:34.370059 12569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:41231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:34059
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:34059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786502224378336-3582-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:34.371234 12569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:34.372308 12569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:34.382328 12576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:34.382467 12569 server_base.cc:1034] running on GCE node
W20260812 02:37:34.383255 12579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:34.383469 12575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:34.384254 12569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:34.385711 12569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:34.386890 12569 hybrid_clock.cc:648] HybridClock initialized: now 1786502254386871 us; error 32 us; skew 500 ppm
I20260812 02:37:34.388878 12569 webserver.cc:460] Webserver started at http://127.3.127.190:35141/ using document root <none> and password file <none>
I20260812 02:37:34.389506 12569 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:34.389573 12569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:34.389775 12569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:34.391865 12569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "c8c343fa5c694b2f995e4cf3ed8c76f5"
format_stamp: "Formatted at 2026-08-12 02:37:34 on dist-test-slave-rmhg"
server_key: "1460376a355ca0293ba87cd96fbabf74"
server_key_iv: "07dd92fa0a3d670d9ec31c67473cdb32"
server_key_version: "encryptionkey@0"
I20260812 02:37:34.392385 12569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "c8c343fa5c694b2f995e4cf3ed8c76f5"
format_stamp: "Formatted at 2026-08-12 02:37:34 on dist-test-slave-rmhg"
server_key: "1460376a355ca0293ba87cd96fbabf74"
server_key_iv: "07dd92fa0a3d670d9ec31c67473cdb32"
server_key_version: "encryptionkey@0"
I20260812 02:37:34.396155 12569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260812 02:37:34.398691 12587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:34.399955 12569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 02:37:34.400067 12569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "c8c343fa5c694b2f995e4cf3ed8c76f5"
format_stamp: "Formatted at 2026-08-12 02:37:34 on dist-test-slave-rmhg"
server_key: "1460376a355ca0293ba87cd96fbabf74"
server_key_iv: "07dd92fa0a3d670d9ec31c67473cdb32"
server_key_version: "encryptionkey@0"
I20260812 02:37:34.400184 12569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:34.417286 12569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:34.418040 12569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:34.418231 12569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:34.424927 12569 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:34059
I20260812 02:37:34.424963 12680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:34059 every 8 connection(s)
I20260812 02:37:34.425953 12569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:37:34.427459 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 12569
I20260812 02:37:34.427567 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:37:34.427826 3582 external_mini_cluster.cc:1428] Setting key 3e4a1d401f768a03118256f34590955e
I20260812 02:37:34.429728 12681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:34.436417 12681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5: Bootstrap starting.
I20260812 02:37:34.437515 12689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:34.429263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.3.127.190:34059)
0812 02:37:34.429922 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:34.429939 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:34.429979 (+ 40us) server_negotiation.cc:407] Connection header received
0812 02:37:34.430629 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:34.430645 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:34.431096 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:34.431403 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:34.431776 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:34.432763 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:34.434005 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:34.434521 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:34.434639 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:34.435563 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:34.435568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:34.435582 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:37:34.436012 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:34.436072 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:37:34.436094 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:34.436172 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:34.436331 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 02:37:34.436613 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":272,"thread_start_us":123,"threads_started":1}
I20260812 02:37:34.438958 12681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:34.439783 12681 log.cc:824] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:34.440476 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:37:34.442404 12681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5: No bootstrap required, opened a new log
I20260812 02:37:34.446166 12681 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c343fa5c694b2f995e4cf3ed8c76f5" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 34059 } }
I20260812 02:37:34.446424 12681 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:34.446460 12681 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8c343fa5c694b2f995e4cf3ed8c76f5, State: Initialized, Role: FOLLOWER
I20260812 02:37:34.446898 12681 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c343fa5c694b2f995e4cf3ed8c76f5" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 34059 } }
I20260812 02:37:34.447029 12681 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:37:34.447072 12681 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:37:34.447145 12681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:34.448402 12681 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c343fa5c694b2f995e4cf3ed8c76f5" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 34059 } }
I20260812 02:37:34.448774 12681 leader_election.cc:302] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8c343fa5c694b2f995e4cf3ed8c76f5; no voters:
I20260812 02:37:34.449102 12681 leader_election.cc:288] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:37:34.449261 12691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:34.449491 12691 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5 [term 1 LEADER]: Becoming Leader. State: Replica: c8c343fa5c694b2f995e4cf3ed8c76f5, State: Running, Role: LEADER
I20260812 02:37:34.449791 12691 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c343fa5c694b2f995e4cf3ed8c76f5" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 34059 } }
I20260812 02:37:34.450398 12681 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:37:34.451613 12693 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8c343fa5c694b2f995e4cf3ed8c76f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c343fa5c694b2f995e4cf3ed8c76f5" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 34059 } } }
I20260812 02:37:34.451836 12693 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:34.451756 12696 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8c343fa5c694b2f995e4cf3ed8c76f5. Latest consensus state: current_term: 1 leader_uuid: "c8c343fa5c694b2f995e4cf3ed8c76f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c343fa5c694b2f995e4cf3ed8c76f5" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 34059 } } }
I20260812 02:37:34.451916 12696 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:34.452383 12708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:37:34.456205 12708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:37:34.462215 12708 catalog_manager.cc:1347] Generated new cluster ID: 76041cd4b5164028a6b0f269405a18f2
I20260812 02:37:34.462309 12708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:37:34.476341 12708 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:37:34.477746 12708 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:37:34.488876 12708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8c343fa5c694b2f995e4cf3ed8c76f5: Generated new TSK 0
I20260812 02:37:34.489845 12708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:37:35.386654 12689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:34.502712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33960 (local address 127.3.127.190:34059)
0812 02:37:34.502858 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:34.502863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:34.503045 (+ 182us) server_negotiation.cc:407] Connection header received
0812 02:37:34.503238 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:34.503241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:34.503293 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:34.503388 (+ 95us) server_negotiation.cc:226] Negotiated authn=JWT
0812 02:37:34.503647 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:34.504275 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:34.505598 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:34.505822 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:34.505904 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:34.506230 (+ 326us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 02:37:35.386219 (+879989us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 02:37:35.386524 (+ 305us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33960: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":53}
W20260812 02:37:35.386741 12689 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33960: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260812 02:37:35.386878 12733 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:34.502628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:34059 (local address 127.0.0.1:33960)
0812 02:37:34.502953 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:34.502969 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:34.503098 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:34.503404 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:34.503407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:34.503415 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0812 02:37:34.503549 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:34.503555 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:34.504396 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:34.504400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:34.505476 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:34.505489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:34.505941 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:34.505945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:34.506038 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:34.506069 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 02:37:35.386496 (+880427us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 02:37:35.386705 (+ 209us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.127.190:34059: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":109,"threads_started":1}
I20260812 02:37:35.387434 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 12569
2026-08-12T02:37:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1185 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (135 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (130 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:37:35 dist-test-slave-rmhg krb5kdc[12747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:37:35 dist-test-slave-rmhg krb5kdc[12747](info): set up 2 sockets
Aug 12 02:37:35 dist-test-slave-rmhg krb5kdc[12747](info): commencing operation
krb5kdc: starting...
W20260812 02:37:36.847635 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:37:36 dist-test-slave-rmhg krb5kdc[12747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:37:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:37:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:37:36.980808 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:38651
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:36921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:38651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:37.101212 12766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:37.101543 12766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:37.101603 12766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:37.105782 12766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:37.105865 12766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:37.105894 12766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:37.105918 12766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:37.105957 12766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:37:37.111739 12766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:36921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:38651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:38651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:37.113103 12766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:37.114269 12766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:37.123178 12772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:37.123183 12776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:37.123236 12773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:37.123391 12766 server_base.cc:1034] running on GCE node
I20260812 02:37:37.124125 12766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:37.125451 12766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:37.126622 12766 hybrid_clock.cc:648] HybridClock initialized: now 1786502257126599 us; error 37 us; skew 500 ppm
Aug 12 02:37:37 dist-test-slave-rmhg krb5kdc[12747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502257, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:37.129416 12766 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:37:37.130383 12766 webserver.cc:460] Webserver started at http://127.3.127.190:41433/ using document root <none> and password file <none>
I20260812 02:37:37.130949 12766 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:37.131016 12766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:37.131239 12766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:37.133419 12766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "313e77c02e3c4cdf8f161bbd5c3cac10"
format_stamp: "Formatted at 2026-08-12 02:37:37 on dist-test-slave-rmhg"
server_key: "80fd2b88b2fa10462272fe1b51bfa525"
server_key_iv: "5407e33889f8e23b663e83e0f449f389"
server_key_version: "encryptionkey@0"
I20260812 02:37:37.134100 12766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "313e77c02e3c4cdf8f161bbd5c3cac10"
format_stamp: "Formatted at 2026-08-12 02:37:37 on dist-test-slave-rmhg"
server_key: "80fd2b88b2fa10462272fe1b51bfa525"
server_key_iv: "5407e33889f8e23b663e83e0f449f389"
server_key_version: "encryptionkey@0"
I20260812 02:37:37.138357 12766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:37:37.141247 12789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:37.142787 12766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:37:37.142944 12766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "313e77c02e3c4cdf8f161bbd5c3cac10"
format_stamp: "Formatted at 2026-08-12 02:37:37 on dist-test-slave-rmhg"
server_key: "80fd2b88b2fa10462272fe1b51bfa525"
server_key_iv: "5407e33889f8e23b663e83e0f449f389"
server_key_version: "encryptionkey@0"
I20260812 02:37:37.143083 12766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:37.203567 12766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:37.206842 12766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:37.207049 12766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:37.214846 12766 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:38651
I20260812 02:37:37.214900 12884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:38651 every 8 connection(s)
I20260812 02:37:37.215936 12766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:37:37.219229 12885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:37.220542 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 12766
I20260812 02:37:37.220672 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:37:37.220993 3582 external_mini_cluster.cc:1428] Setting key aad701a298d03a6c0858d4317b958f0f
I20260812 02:37:37.226217 12885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10: Bootstrap starting.
I20260812 02:37:37.228770 12885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:37.229737 12885 log.cc:824] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:37:37 dist-test-slave-rmhg krb5kdc[12747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:37.232366 12885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10: No bootstrap required, opened a new log
I20260812 02:37:37.236706 12885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313e77c02e3c4cdf8f161bbd5c3cac10" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38651 } }
I20260812 02:37:37.236999 12885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:37.237053 12885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313e77c02e3c4cdf8f161bbd5c3cac10, State: Initialized, Role: FOLLOWER
I20260812 02:37:37.237598 12891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:37.222615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37292 (local address 127.3.127.190:38651)
0812 02:37:37.223162 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:37.223188 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:37.223415 (+ 227us) server_negotiation.cc:407] Connection header received
0812 02:37:37.224294 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:37.224317 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:37.224739 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:37.225101 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:37.226068 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:37.227169 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:37.228081 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:37.228610 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:37.228785 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:37.231955 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:37.231961 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:37.231977 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:37.232011 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:37.233848 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:37.234468 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:37.234474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:37.234480 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:37.234667 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:37.235012 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:37.235015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:37.235016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:37.235519 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:37.235764 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:37.236164 (+ 400us) server_negotiation.cc:299] Negotiation successful
0812 02:37:37.236591 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":144,"threads_started":1}
I20260812 02:37:37.237613 12885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313e77c02e3c4cdf8f161bbd5c3cac10" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38651 } }
I20260812 02:37:37.237815 12885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:37:37.237880 12885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:37:37.237987 12885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:37.239166 12885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313e77c02e3c4cdf8f161bbd5c3cac10" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38651 } }
I20260812 02:37:37.239596 12885 leader_election.cc:302] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 313e77c02e3c4cdf8f161bbd5c3cac10; no voters:
I20260812 02:37:37.239984 12885 leader_election.cc:288] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:37:37.240140 12896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:37.240382 12896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10 [term 1 LEADER]: Becoming Leader. State: Replica: 313e77c02e3c4cdf8f161bbd5c3cac10, State: Running, Role: LEADER
I20260812 02:37:37.240797 12896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313e77c02e3c4cdf8f161bbd5c3cac10" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38651 } }
I20260812 02:37:37.241343 12885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:37:37.242273 12896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 313e77c02e3c4cdf8f161bbd5c3cac10. Latest consensus state: current_term: 1 leader_uuid: "313e77c02e3c4cdf8f161bbd5c3cac10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313e77c02e3c4cdf8f161bbd5c3cac10" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38651 } } }
I20260812 02:37:37.242405 12896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:37.242516 12898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "313e77c02e3c4cdf8f161bbd5c3cac10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313e77c02e3c4cdf8f161bbd5c3cac10" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38651 } } }
I20260812 02:37:37.242621 12898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:37.242944 12912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:37:37.246098 12912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:37:37.252381 12912 catalog_manager.cc:1347] Generated new cluster ID: 82499a01fe8142f39f97863a3e3ae8a8
I20260812 02:37:37.252470 12912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:37:37.258282 12912 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:37:37.259238 12912 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:37:37.263895 12912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 313e77c02e3c4cdf8f161bbd5c3cac10: Generated new TSK 0
I20260812 02:37:37.264794 12912 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
I20260812 02:37:37.306754 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38651
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:36921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:37.460937 12933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:37.461267 12933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:37.461314 12933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:37.465571 12933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:37.465657 12933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:37.465724 12933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:37:37.471428 12933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:36921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:37.472638 12933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:37.473778 12933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:37.484723 12947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:37.484889 12933 server_base.cc:1034] running on GCE node
W20260812 02:37:37.484743 12944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:37.485103 12943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:37.485615 12933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:37.486320 12933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:37.487499 12933 hybrid_clock.cc:648] HybridClock initialized: now 1786502257487488 us; error 43 us; skew 500 ppm
Aug 12 02:37:37 dist-test-slave-rmhg krb5kdc[12747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502257, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:37.490656 12933 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:37:37.491695 12933 webserver.cc:460] Webserver started at http://127.3.127.129:44715/ using document root <none> and password file <none>
I20260812 02:37:37.492295 12933 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:37.492367 12933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:37.492587 12933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:37.494863 12933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "be1478f6fc72429b85ef2daf8657e48c"
format_stamp: "Formatted at 2026-08-12 02:37:37 on dist-test-slave-rmhg"
server_key: "f6d824603b2ba2da5815782d00d8407d"
server_key_iv: "75b5d4395ef67071d447f2307051c494"
server_key_version: "encryptionkey@0"
I20260812 02:37:37.495440 12933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "be1478f6fc72429b85ef2daf8657e48c"
format_stamp: "Formatted at 2026-08-12 02:37:37 on dist-test-slave-rmhg"
server_key: "f6d824603b2ba2da5815782d00d8407d"
server_key_iv: "75b5d4395ef67071d447f2307051c494"
server_key_version: "encryptionkey@0"
I20260812 02:37:37.499558 12933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 02:37:37.502342 12960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:37.503846 12933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:37.504010 12933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "be1478f6fc72429b85ef2daf8657e48c"
format_stamp: "Formatted at 2026-08-12 02:37:37 on dist-test-slave-rmhg"
server_key: "f6d824603b2ba2da5815782d00d8407d"
server_key_iv: "75b5d4395ef67071d447f2307051c494"
server_key_version: "encryptionkey@0"
I20260812 02:37:37.504135 12933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:37.518894 12933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:37.522507 12933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:37.522716 12933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:37.523475 12933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:37.524508 12933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:37.524560 12933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:37.524601 12933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:37.524616 12933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:37.533865 12933 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:38117
I20260812 02:37:37.533890 13147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:38117 every 8 connection(s)
I20260812 02:37:37.534873 12933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:37:37.536940 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 12933
I20260812 02:37:37.537055 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:37:37.537386 3582 external_mini_cluster.cc:1428] Setting key dcf20e4a110188f0723f52072af26a57
Aug 12 02:37:37 dist-test-slave-rmhg krb5kdc[12747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502257, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.127.130@KRBTEST.COM; defaulting to no policy
I20260812 02:37:37.552049 12891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:37.536769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:37233 (local address 127.3.127.190:38651)
0812 02:37:37.536937 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:37.536942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:37.538023 (+ 1081us) server_negotiation.cc:407] Connection header received
0812 02:37:37.539861 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:37.539864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:37.539913 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:37.539984 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:37.541767 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:37.542406 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:37.544152 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:37.544436 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:37.544539 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:37.548087 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:37.548091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:37.548093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:37.548123 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:37.549566 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:37.550235 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:37.550238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:37.550240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:37.550330 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:37.550739 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:37.550743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:37.550744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:37.550992 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:37.551079 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:37.551754 (+ 675us) server_negotiation.cc:299] Negotiation successful
0812 02:37:37.551891 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.130@KRBTEST.COM" created.
I20260812 02:37:37.553166 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:37.537284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38651 (local address 127.3.127.129:37233)
0812 02:37:37.537855 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:37.537906 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:37.539598 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:37.540241 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:37.540252 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:37.540772 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:37.541569 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:37.541592 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:37.542547 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:37.542550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:37.544002 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:37.544017 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:37.544630 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:37.544634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:37.544869 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:37.545683 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:37.545710 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:37.547846 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:37.549701 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:37.549709 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:37.549732 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:37.550090 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:37.550407 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:37.550409 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:37.550410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:37.550580 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:37.551131 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:37.551137 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:37.551428 (+ 291us) client_negotiation.cc:769] Sending connection context
0812 02:37:37.551625 (+ 197us) client_negotiation.cc:240] Negotiation successful
0812 02:37:37.551935 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":148,"threads_started":1}
I20260812 02:37:37.554625 13148 heartbeater.cc:347] Connected to a master server at 127.3.127.190:38651
I20260812 02:37:37.554873 13148 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:37.555455 13148 heartbeater.cc:511] Master 127.3.127.190:38651 requested a full tablet report, sending...
I20260812 02:37:37.557128 12825 ts_manager.cc:194] Registered new tserver with Master: be1478f6fc72429b85ef2daf8657e48c (127.3.127.129:38117)
I20260812 02:37:37.558354 12825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:37233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
I20260812 02:37:37.572521 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38651
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:36921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:37.696105 13158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:37.696406 13158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:37.696450 13158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:37.700608 13158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:37.700685 13158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:37.700747 13158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:37:37.706400 13158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:36921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:37.707633 13158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:37.708724 13158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:37.719343 13168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:37.719370 13158 server_base.cc:1034] running on GCE node
W20260812 02:37:37.719359 13169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:37.719388 13171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:37.720170 13158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:37.720845 13158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:37.722030 13158 hybrid_clock.cc:648] HybridClock initialized: now 1786502257721995 us; error 52 us; skew 500 ppm
Aug 12 02:37:37 dist-test-slave-rmhg krb5kdc[12747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502257, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:37.724984 13158 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:37:37.726004 13158 webserver.cc:460] Webserver started at http://127.3.127.130:44207/ using document root <none> and password file <none>
I20260812 02:37:37.726572 13158 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:37.726631 13158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:37.726819 13158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:37.728914 13158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "63971128dbf44b029ffdf3eec047b568"
format_stamp: "Formatted at 2026-08-12 02:37:37 on dist-test-slave-rmhg"
server_key: "0772dfaee79b1c241440de94285016b6"
server_key_iv: "a4bc3ce9c09506af68ba9a8df5180faf"
server_key_version: "encryptionkey@0"
I20260812 02:37:37.729506 13158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "63971128dbf44b029ffdf3eec047b568"
format_stamp: "Formatted at 2026-08-12 02:37:37 on dist-test-slave-rmhg"
server_key: "0772dfaee79b1c241440de94285016b6"
server_key_iv: "a4bc3ce9c09506af68ba9a8df5180faf"
server_key_version: "encryptionkey@0"
I20260812 02:37:37.733558 13158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:37:37.736114 13184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:37.737498 13158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:37.737775 13158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "63971128dbf44b029ffdf3eec047b568"
format_stamp: "Formatted at 2026-08-12 02:37:37 on dist-test-slave-rmhg"
server_key: "0772dfaee79b1c241440de94285016b6"
server_key_iv: "a4bc3ce9c09506af68ba9a8df5180faf"
server_key_version: "encryptionkey@0"
I20260812 02:37:37.737900 13158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:37.766649 13158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:37.769959 13158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:37.770138 13158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:37.770767 13158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:37.771683 13158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:37.771734 13158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:37.771775 13158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:37.771795 13158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:37.781217 13158 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:35745
I20260812 02:37:37.781301 13366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:35745 every 8 connection(s)
I20260812 02:37:37.782285 13158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:37:37.791265 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 13158
I20260812 02:37:37.791381 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:37:37.791661 3582 external_mini_cluster.cc:1428] Setting key 2d58f584cdb1360e3e6af4be027a3c9c
Aug 12 02:37:37 dist-test-slave-rmhg krb5kdc[12747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502257, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:37.797511 12891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:37.784271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:37389 (local address 127.3.127.190:38651)
0812 02:37:37.784406 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:37.784410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:37.785421 (+ 1011us) server_negotiation.cc:407] Connection header received
0812 02:37:37.786892 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:37.786896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:37.786960 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:37.787039 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:37.788434 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:37.788977 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:37.790261 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:37.790504 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:37.790603 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:37.793606 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:37.793610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:37.793612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:37.793640 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:37.795189 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:37.795804 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:37.795807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:37.795809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:37.795895 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:37.796272 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:37.796275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:37.796276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:37.796485 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:37.796581 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:37.797195 (+ 614us) server_negotiation.cc:299] Negotiation successful
0812 02:37:37.797342 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 02:37:37.798434 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:37.784697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38651 (local address 127.3.127.130:37389)
0812 02:37:37.785254 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:37.785306 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:37.786642 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:37.787249 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:37.787257 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:37.787705 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:37.788261 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:37.788277 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:37.789121 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:37.789124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:37.790120 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:37.790131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:37.790679 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:37.790684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:37.790871 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:37.791578 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:37.791598 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:37.793385 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:37.795333 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:37.795342 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:37.795362 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:37.795679 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:37.795986 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:37.795989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:37.795991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:37.796159 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:37.796598 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:37.796606 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:37.796899 (+ 293us) client_negotiation.cc:769] Sending connection context
0812 02:37:37.797084 (+ 185us) client_negotiation.cc:240] Negotiation successful
0812 02:37:37.797528 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":139,"threads_started":1}
I20260812 02:37:37.799734 13367 heartbeater.cc:347] Connected to a master server at 127.3.127.190:38651
I20260812 02:37:37.800040 13367 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:37.800729 13367 heartbeater.cc:511] Master 127.3.127.190:38651 requested a full tablet report, sending...
I20260812 02:37:37.802090 12825 ts_manager.cc:194] Registered new tserver with Master: 63971128dbf44b029ffdf3eec047b568 (127.3.127.130:35745)
WARNING: no policy specified for kudu/127.3.127.131@KRBTEST.COM; defaulting to no policy
I20260812 02:37:37.802820 12825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:37389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
I20260812 02:37:37.821698 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38651
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:36921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:37.939091 13374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:37.939369 13374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:37.939412 13374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:37.943684 13374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:37.943761 13374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:37.943832 13374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:37:37.949679 13374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:36921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:37.950863 13374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:37.952024 13374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:37.962675 13384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:37.962689 13379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:37.962796 13374 server_base.cc:1034] running on GCE node
W20260812 02:37:37.962689 13380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:37.964309 13374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:37.965104 13374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:37.966308 13374 hybrid_clock.cc:648] HybridClock initialized: now 1786502257966287 us; error 35 us; skew 500 ppm
Aug 12 02:37:37 dist-test-slave-rmhg krb5kdc[12747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502257, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:37.969354 13374 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:37:37.970325 13374 webserver.cc:460] Webserver started at http://127.3.127.131:38845/ using document root <none> and password file <none>
I20260812 02:37:37.970886 13374 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:37.970952 13374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:37.971179 13374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:37.973387 13374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "b80f211511e9426bb24f638765d631d8"
format_stamp: "Formatted at 2026-08-12 02:37:37 on dist-test-slave-rmhg"
server_key: "4d8238be3ed6a3ff03a98e3549259a80"
server_key_iv: "24e5f83a493f372951afb476e0eefd8c"
server_key_version: "encryptionkey@0"
I20260812 02:37:37.973951 13374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "b80f211511e9426bb24f638765d631d8"
format_stamp: "Formatted at 2026-08-12 02:37:37 on dist-test-slave-rmhg"
server_key: "4d8238be3ed6a3ff03a98e3549259a80"
server_key_iv: "24e5f83a493f372951afb476e0eefd8c"
server_key_version: "encryptionkey@0"
I20260812 02:37:37.977990 13374 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:37:37.980782 13393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:37.982193 13374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:37:37.982321 13374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "b80f211511e9426bb24f638765d631d8"
format_stamp: "Formatted at 2026-08-12 02:37:37 on dist-test-slave-rmhg"
server_key: "4d8238be3ed6a3ff03a98e3549259a80"
server_key_iv: "24e5f83a493f372951afb476e0eefd8c"
server_key_version: "encryptionkey@0"
I20260812 02:37:37.982434 13374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:37.997498 13374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:38.000841 13374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:38.001049 13374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:38.001729 13374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:38.002671 13374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:38.002722 13374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:38.002775 13374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:38.002802 13374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:38.012068 13374 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:39763
I20260812 02:37:38.012059 13585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:39763 every 8 connection(s)
I20260812 02:37:38.013134 13374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:37:38.019541 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 13374
I20260812 02:37:38.019665 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:37:38.019941 3582 external_mini_cluster.cc:1428] Setting key 67a8129414fc89d52983a41f630fb0aa
Aug 12 02:37:38 dist-test-slave-rmhg krb5kdc[12747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502257, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:38.027972 12891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:38.015195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:53061 (local address 127.3.127.190:38651)
0812 02:37:38.015340 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:38.015344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:38.016302 (+ 958us) server_negotiation.cc:407] Connection header received
0812 02:37:38.017811 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:38.017815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:38.017877 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:38.017954 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:38.019263 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:38.019852 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:38.021258 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:38.021553 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:38.021643 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:38.024365 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:38.024369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:38.024371 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:38.024399 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:38.025871 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:38.026437 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:38.026440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:38.026442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:38.026575 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:38.026912 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:38.026915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:38.026917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:38.027128 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:38.027221 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:38.027712 (+ 491us) server_negotiation.cc:299] Negotiation successful
0812 02:37:38.027829 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 02:37:38.028836 13588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:38.015612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38651 (local address 127.3.127.131:53061)
0812 02:37:38.016132 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:38.016179 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:38.017576 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:38.018175 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:38.018183 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:38.018578 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:38.019111 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:38.019126 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:38.019978 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:38.019980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:38.021093 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:38.021107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:38.021720 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:38.021724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:38.021944 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:38.022644 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:38.022661 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:38.024162 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:38.025991 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:38.025997 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:38.026012 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:38.026331 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:38.026650 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:38.026652 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:38.026653 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:38.026807 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:38.027218 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:38.027223 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:38.027484 (+ 261us) client_negotiation.cc:769] Sending connection context
0812 02:37:38.027652 (+ 168us) client_negotiation.cc:240] Negotiation successful
0812 02:37:38.027952 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":137,"threads_started":1}
I20260812 02:37:38.030076 13586 heartbeater.cc:347] Connected to a master server at 127.3.127.190:38651
I20260812 02:37:38.030346 13586 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:38.030905 13586 heartbeater.cc:511] Master 127.3.127.190:38651 requested a full tablet report, sending...
I20260812 02:37:38.032037 12825 ts_manager.cc:194] Registered new tserver with Master: b80f211511e9426bb24f638765d631d8 (127.3.127.131:39763)
I20260812 02:37:38.032752 12825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:53061
I20260812 02:37:38.034180 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/test-admin.keytab.
Aug 12 02:37:38 dist-test-slave-rmhg krb5kdc[12747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:38.053332 3582 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 02:37:38.056433 13601 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:38.054943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55272 (local address 127.3.127.129:38117)
0812 02:37:38.055313 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:38.055320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:38.055345 (+ 25us) server_negotiation.cc:407] Connection header received
0812 02:37:38.055398 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:38.055403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:38.055595 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:38.055723 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:38.056264 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55272: BlockingRecv error: recv got EOF from 127.0.0.1:55272 (error 108)
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":135,"threads_started":1}
W20260812 02:37:38.056489 3582 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/krb5cc' of type FILE
Aug 12 02:37:38 dist-test-slave-rmhg krb5kdc[12747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:38.057561 3582 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 02:37:38 dist-test-slave-rmhg krb5kdc[12747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:38.058501 3582 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 02:37:38 dist-test-slave-rmhg krb5kdc[12747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.129@KRBTEST.COM
I20260812 02:37:38.068687 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:38.059460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55286 (local address 127.3.127.129:38117)
0812 02:37:38.059617 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:38.059621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:38.059638 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:37:38.059794 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:38.059797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:38.059842 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:38.059908 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:38.060392 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:38.061249 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:38.062098 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:38.062493 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:38.062574 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:38.064842 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:38.064857 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:38.064875 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:38.064909 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:38.066588 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:38.067067 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:38.067074 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:38.067075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:38.067172 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:38.067604 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:38.067607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:38.067608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:38.067919 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:38.068117 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:38.068337 (+ 220us) server_negotiation.cc:299] Negotiation successful
0812 02:37:38.068516 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260812 02:37:38.073058 13601 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:38.072034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55296 (local address 127.3.127.129:38117)
0812 02:37:38.072146 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:38.072150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:38.072319 (+ 169us) server_negotiation.cc:407] Connection header received
0812 02:37:38.072425 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:38.072428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:38.072468 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:38.072533 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:38.072958 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55296: BlockingRecv error: recv got EOF from 127.0.0.1:55296 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260812 02:37:38.073277 3582 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/krb5cc' of type FILE
Aug 12 02:37:38 dist-test-slave-rmhg krb5kdc[12747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:38.074462 3582 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 02:37:38 dist-test-slave-rmhg krb5kdc[12747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:38.075481 3582 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 02:37:38 dist-test-slave-rmhg krb5kdc[12747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.129@KRBTEST.COM
I20260812 02:37:38.085398 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:38.076489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55300 (local address 127.3.127.129:38117)
0812 02:37:38.076635 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:38.076639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:38.076753 (+ 114us) server_negotiation.cc:407] Connection header received
0812 02:37:38.076859 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:38.076861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:38.076909 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:38.077020 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:38.077774 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:38.078330 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:38.079188 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:38.079417 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:38.079503 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:38.082096 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:38.082101 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:38.082103 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:38.082135 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:38.083562 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:38.084036 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:38.084040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:38.084041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:38.084138 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:38.084456 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:38.084458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:38.084469 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:38.084675 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:38.084799 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:38.085030 (+ 231us) server_negotiation.cc:299] Negotiation successful
0812 02:37:38.085234 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260812 02:37:38.088222 13601 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:38.087319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55308 (local address 127.3.127.129:38117)
0812 02:37:38.087474 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:38.087479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:38.087559 (+ 80us) server_negotiation.cc:407] Connection header received
0812 02:37:38.087649 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:38.087652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:38.087693 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:38.087755 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:38.088090 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55308: BlockingRecv error: recv got EOF from 127.0.0.1:55308 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260812 02:37:38.088587 3582 init.cc:244] Error while opening credential cache '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786502224378336-3582-0/krb5kdc/krb5cc)
Aug 12 02:37:38 dist-test-slave-rmhg krb5kdc[12747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:38.089720 3582 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 02:37:38 dist-test-slave-rmhg krb5kdc[12747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:38.090668 3582 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 02:37:38 dist-test-slave-rmhg krb5kdc[12747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.129@KRBTEST.COM
I20260812 02:37:38.099397 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:38.091632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55312 (local address 127.3.127.129:38117)
0812 02:37:38.091759 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:38.091764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:38.091830 (+ 66us) server_negotiation.cc:407] Connection header received
0812 02:37:38.091973 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:38.091975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:38.092015 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:38.092081 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:38.092629 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:38.093216 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:38.094033 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:38.094265 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:38.094356 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:38.096638 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:38.096641 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:38.096643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:38.096671 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:38.097874 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:38.098313 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:38.098316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:38.098318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:38.098407 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:38.098660 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:38.098664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:38.098665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:38.098877 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:38.099038 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:38.099159 (+ 121us) server_negotiation.cc:299] Negotiation successful
0812 02:37:38.099270 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 02:37:38.100586 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 12933
I20260812 02:37:38.117034 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 13158
I20260812 02:37:38.130388 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 13374
I20260812 02:37:38.176388 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 12766
2026-08-12T02:37:38Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2524 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:37:38 dist-test-slave-rmhg krb5kdc[13623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:37:38 dist-test-slave-rmhg krb5kdc[13623](info): set up 2 sockets
Aug 12 02:37:38 dist-test-slave-rmhg krb5kdc[13623](info): commencing operation
krb5kdc: starting...
W20260812 02:37:39.360158 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:37:39 dist-test-slave-rmhg krb5kdc[13623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:37:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:37:39Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.190.keytab.
Entry for principal oryx/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.190.keytab.
I20260812 02:37:39.490206 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:38495
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.190.keytab
--principal=oryx/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:38495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:39.606932 13647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:39.607224 13647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:39.607285 13647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:39.611395 13647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:39.611471 13647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:39.611498 13647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:39.611519 13647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:39.611548 13647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:37:39.617000 13647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:38495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.190.keytab
--principal=oryx/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:38495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:39.618157 13647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:39.619216 13647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:39.627911 13655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:39.627920 13656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:39.628099 13647 server_base.cc:1034] running on GCE node
W20260812 02:37:39.627918 13659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:39.628854 13647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:39.630062 13647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:39.631228 13647 hybrid_clock.cc:648] HybridClock initialized: now 1786502259631214 us; error 28 us; skew 500 ppm
Aug 12 02:37:39 dist-test-slave-rmhg krb5kdc[13623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502259, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:39.634047 13647 init.cc:377] Logged in from keytab as oryx/127.3.127.190@KRBTEST.COM (short username oryx)
I20260812 02:37:39.635011 13647 webserver.cc:460] Webserver started at http://127.3.127.190:38497/ using document root <none> and password file <none>
I20260812 02:37:39.635581 13647 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:39.635651 13647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:39.635851 13647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:39.638018 13647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "05763676674c4a82b5e510ce078c3b07"
format_stamp: "Formatted at 2026-08-12 02:37:39 on dist-test-slave-rmhg"
server_key: "6b44d09def1b9e4abc81dea56f0e382d"
server_key_iv: "7fc90024961258688d576cecaad18adf"
server_key_version: "encryptionkey@0"
I20260812 02:37:39.638674 13647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "05763676674c4a82b5e510ce078c3b07"
format_stamp: "Formatted at 2026-08-12 02:37:39 on dist-test-slave-rmhg"
server_key: "6b44d09def1b9e4abc81dea56f0e382d"
server_key_iv: "7fc90024961258688d576cecaad18adf"
server_key_version: "encryptionkey@0"
I20260812 02:37:39.642656 13647 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:37:39.645501 13671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:39.646920 13647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:39.647076 13647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "05763676674c4a82b5e510ce078c3b07"
format_stamp: "Formatted at 2026-08-12 02:37:39 on dist-test-slave-rmhg"
server_key: "6b44d09def1b9e4abc81dea56f0e382d"
server_key_iv: "7fc90024961258688d576cecaad18adf"
server_key_version: "encryptionkey@0"
I20260812 02:37:39.647195 13647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:39.663384 13647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:39.666725 13647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:39.666921 13647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:39.674762 13647 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:38495
I20260812 02:37:39.674965 13747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:38495 every 8 connection(s)
I20260812 02:37:39.676100 13647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:37:39.677977 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 13647
I20260812 02:37:39.678117 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:37:39.678443 3582 external_mini_cluster.cc:1428] Setting key 416efab7c531b46096abf48f45241207
I20260812 02:37:39.679531 13748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:39.686846 13748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07: Bootstrap starting.
Aug 12 02:37:39 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM
I20260812 02:37:39.689755 13748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:39.690522 13748 log.cc:824] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:39.692757 13748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07: No bootstrap required, opened a new log
I20260812 02:37:39.694572 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:39.680007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.3.127.190:38495)
0812 02:37:39.680500 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:39.680517 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:39.681561 (+ 1044us) server_negotiation.cc:407] Connection header received
0812 02:37:39.682342 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:39.682363 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:39.682727 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:39.683022 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:39.683965 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:39.685057 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:39.686233 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:39.686730 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:39.686853 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:39.689714 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:39.689719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:39.689734 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:39.689766 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:39.691580 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:39.692079 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:39.692083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:39.692088 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:39.692211 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:39.692528 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:39.692532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:39.692533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:39.693001 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:39.693210 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:39.693493 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 02:37:39.693790 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":148,"threads_started":1}
I20260812 02:37:39.696955 13748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } }
I20260812 02:37:39.697223 13748 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:39.697286 13748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05763676674c4a82b5e510ce078c3b07, State: Initialized, Role: FOLLOWER
I20260812 02:37:39.697707 13748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } }
I20260812 02:37:39.697839 13748 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:37:39.697903 13748 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:37:39.698006 13748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:39.698954 13748 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } }
I20260812 02:37:39.699302 13748 leader_election.cc:302] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05763676674c4a82b5e510ce078c3b07; no voters:
I20260812 02:37:39.699615 13748 leader_election.cc:288] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:37:39.699723 13757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:39.699967 13757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [term 1 LEADER]: Becoming Leader. State: Replica: 05763676674c4a82b5e510ce078c3b07, State: Running, Role: LEADER
I20260812 02:37:39.700307 13757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } }
I20260812 02:37:39.700933 13748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:37:39.702246 13762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05763676674c4a82b5e510ce078c3b07. Latest consensus state: current_term: 1 leader_uuid: "05763676674c4a82b5e510ce078c3b07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } }
I20260812 02:37:39.702253 13761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05763676674c4a82b5e510ce078c3b07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } }
I20260812 02:37:39.702418 13761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:39.702418 13762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:39.703080 13777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:37:39.706346 13777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:37:39.711726 13777 catalog_manager.cc:1347] Generated new cluster ID: 1edca9d490854f83b1c360530e4749c0
I20260812 02:37:39.711812 13777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:37:39.721407 13777 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:37:39.722402 13777 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:37:39.731711 13777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07: Generated new TSK 0
I20260812 02:37:39.732496 13777 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.129.keytab.
Entry for principal oryx/127.3.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.129.keytab.
I20260812 02:37:39.764603 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38495
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.129.keytab
--principal=oryx/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:39.885993 13794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:39.886307 13794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:39.886373 13794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:39.890671 13794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:39.890760 13794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:39.890849 13794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:37:39.896471 13794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.129.keytab
--principal=oryx/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:39.897737 13794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:39.899044 13794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:39.909520 13805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:39.909698 13803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:39.909821 13802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:39.909795 13794 server_base.cc:1034] running on GCE node
I20260812 02:37:39.910332 13794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:39.911049 13794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:39.912212 13794 hybrid_clock.cc:648] HybridClock initialized: now 1786502259912193 us; error 33 us; skew 500 ppm
Aug 12 02:37:39 dist-test-slave-rmhg krb5kdc[13623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502259, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:39.915215 13794 init.cc:377] Logged in from keytab as oryx/127.3.127.129@KRBTEST.COM (short username oryx)
I20260812 02:37:39.916285 13794 webserver.cc:460] Webserver started at http://127.3.127.129:39403/ using document root <none> and password file <none>
I20260812 02:37:39.916939 13794 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:39.917097 13794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:39.917408 13794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:39.919700 13794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "430e867652ed41ff8cda09459a710434"
format_stamp: "Formatted at 2026-08-12 02:37:39 on dist-test-slave-rmhg"
server_key: "7323242eca7db23784291bf347ff5d91"
server_key_iv: "e8e27b3635d754102e77e68e7b1fcfd6"
server_key_version: "encryptionkey@0"
I20260812 02:37:39.920316 13794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "430e867652ed41ff8cda09459a710434"
format_stamp: "Formatted at 2026-08-12 02:37:39 on dist-test-slave-rmhg"
server_key: "7323242eca7db23784291bf347ff5d91"
server_key_iv: "e8e27b3635d754102e77e68e7b1fcfd6"
server_key_version: "encryptionkey@0"
I20260812 02:37:39.924412 13794 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:37:39.927402 13816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:39.928838 13794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 02:37:39.929003 13794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "430e867652ed41ff8cda09459a710434"
format_stamp: "Formatted at 2026-08-12 02:37:39 on dist-test-slave-rmhg"
server_key: "7323242eca7db23784291bf347ff5d91"
server_key_iv: "e8e27b3635d754102e77e68e7b1fcfd6"
server_key_version: "encryptionkey@0"
I20260812 02:37:39.929131 13794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:39.956919 13794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:39.960772 13794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:39.960999 13794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:39.961742 13794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:39.962730 13794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:39.962782 13794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:39.962842 13794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:39.962870 13794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:39.972334 13794 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:37785
I20260812 02:37:39.972357 14003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:37785 every 8 connection(s)
I20260812 02:37:39.973481 13794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:37:39.983109 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 13794
I20260812 02:37:39.983232 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:37:39.983498 3582 external_mini_cluster.cc:1428] Setting key 59090e04e057981dae0331d96dd577bb
Aug 12 02:37:39 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502259, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM
I20260812 02:37:39.988534 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:39.975561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:56783 (local address 127.3.127.190:38495)
0812 02:37:39.975745 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:39.975751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:39.976993 (+ 1242us) server_negotiation.cc:407] Connection header received
0812 02:37:39.978075 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:39.978078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:39.978150 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:39.978235 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:39.979667 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:39.980245 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:39.981638 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:39.981901 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:39.981996 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:39.984871 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:39.984875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:39.984878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:39.984909 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:39.986312 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:39.986883 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:39.986886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:39.986887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:39.986970 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:39.987326 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:39.987330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:39.987332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:39.987534 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:39.987637 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:39.988218 (+ 581us) server_negotiation.cc:299] Negotiation successful
0812 02:37:39.988363 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 02:37:39.989696 14008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:39.975998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.129:56783)
0812 02:37:39.976804 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:39.976858 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:39.977812 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:39.978421 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:39.978430 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:39.978877 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:39.979459 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:39.979475 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:39.980415 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:39.980418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:39.981493 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:39.981505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:39.982068 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:39.982071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:39.982260 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:39.982885 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:39.982903 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:39.984684 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:39.986449 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:39.986456 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:39.986473 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:39.986778 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:39.987056 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:39.987059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:39.987062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:39.987232 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:39.987655 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:39.987663 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:39.987952 (+ 289us) client_negotiation.cc:769] Sending connection context
0812 02:37:39.988153 (+ 201us) client_negotiation.cc:240] Negotiation successful
0812 02:37:39.988537 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":145,"threads_started":1}
I20260812 02:37:39.991180 14005 heartbeater.cc:347] Connected to a master server at 127.3.127.190:38495
I20260812 02:37:39.991456 14005 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:39.992025 14005 heartbeater.cc:511] Master 127.3.127.190:38495 requested a full tablet report, sending...
I20260812 02:37:39.993827 13692 ts_manager.cc:194] Registered new tserver with Master: 430e867652ed41ff8cda09459a710434 (127.3.127.129:37785)
WARNING: no policy specified for oryx/127.3.127.130@KRBTEST.COM; defaulting to no policy
I20260812 02:37:39.995069 13692 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:56783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.127.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.130.keytab.
Entry for principal oryx/127.3.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.130.keytab.
I20260812 02:37:40.016649 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38495
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.130.keytab
--principal=oryx/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:40.137483 14013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:40.137877 14013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:40.137948 14013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:40.142367 14013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:40.142463 14013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:40.142563 14013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:37:40.148433 14013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.130.keytab
--principal=oryx/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:40.149863 14013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:40.151008 14013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:37:40.161425 14013 server_base.cc:1034] running on GCE node
W20260812 02:37:40.161412 14021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:40.161414 14020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:40.161664 14023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:40.162073 14013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:40.162750 14013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:40.163915 14013 hybrid_clock.cc:648] HybridClock initialized: now 1786502260163885 us; error 41 us; skew 500 ppm
Aug 12 02:37:40 dist-test-slave-rmhg krb5kdc[13623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502260, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:40.166766 14013 init.cc:377] Logged in from keytab as oryx/127.3.127.130@KRBTEST.COM (short username oryx)
I20260812 02:37:40.167730 14013 webserver.cc:460] Webserver started at http://127.3.127.130:46649/ using document root <none> and password file <none>
I20260812 02:37:40.168287 14013 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:40.168351 14013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:40.168529 14013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:40.170717 14013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "3737fe1f65fa40a484cc4f7dc3e030ba"
format_stamp: "Formatted at 2026-08-12 02:37:40 on dist-test-slave-rmhg"
server_key: "25ed4d997c779efd189a5798b1216215"
server_key_iv: "4430d0037ca540e5a4e364420ec846ec"
server_key_version: "encryptionkey@0"
I20260812 02:37:40.171262 14013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "3737fe1f65fa40a484cc4f7dc3e030ba"
format_stamp: "Formatted at 2026-08-12 02:37:40 on dist-test-slave-rmhg"
server_key: "25ed4d997c779efd189a5798b1216215"
server_key_iv: "4430d0037ca540e5a4e364420ec846ec"
server_key_version: "encryptionkey@0"
I20260812 02:37:40.175348 14013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:37:40.178107 14035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:40.179421 14013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:40.179533 14013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "3737fe1f65fa40a484cc4f7dc3e030ba"
format_stamp: "Formatted at 2026-08-12 02:37:40 on dist-test-slave-rmhg"
server_key: "25ed4d997c779efd189a5798b1216215"
server_key_iv: "4430d0037ca540e5a4e364420ec846ec"
server_key_version: "encryptionkey@0"
I20260812 02:37:40.179638 14013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:40.216498 14013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:40.220433 14013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:40.220671 14013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:40.221504 14013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:40.222640 14013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:40.222718 14013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:40.222790 14013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:40.222816 14013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:40.234781 14013 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:34357
I20260812 02:37:40.234903 14232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:34357 every 8 connection(s)
I20260812 02:37:40.236034 14013 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:37:40.245652 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 14013
I20260812 02:37:40.245788 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:37:40.246121 3582 external_mini_cluster.cc:1428] Setting key 0fc767b3565db4d732b07db29b0b483f
Aug 12 02:37:40 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502260, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.3.127.131@KRBTEST.COM; defaulting to no policy
I20260812 02:37:40.259771 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:40.241400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:52141 (local address 127.3.127.190:38495)
0812 02:37:40.244920 (+ 3520us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:40.244926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:40.244947 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:37:40.244999 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:40.245002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:40.245059 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:40.245198 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:40.248451 (+ 3253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.249334 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.251247 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.251651 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.251767 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:40.255222 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:40.255227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:40.255230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:40.255265 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:40.257079 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:40.257864 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:40.257868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:40.257870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:40.257978 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:40.258443 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:40.258447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:40.258448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:40.258699 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:40.258791 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:40.259506 (+ 715us) server_negotiation.cc:299] Negotiation successful
0812 02:37:40.259637 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3396}
I20260812 02:37:40.260823 14236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:40.238897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.130:52141)
0812 02:37:40.241027 (+ 2130us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:40.241103 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:40.242320 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:40.246765 (+ 4445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:40.246779 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:40.247411 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:40.248272 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:40.248301 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.249517 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.249520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:40.251093 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:40.251109 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.251874 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.251879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:40.252127 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:40.252889 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:40.252915 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:40.254971 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:40.257292 (+ 2321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:40.257298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:40.257314 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:40.257709 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:40.258073 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:40.258076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:40.258078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:40.258279 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:40.258805 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:40.258813 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:40.259237 (+ 424us) client_negotiation.cc:769] Sending connection context
0812 02:37:40.259427 (+ 190us) client_negotiation.cc:240] Negotiation successful
0812 02:37:40.259738 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1820,"thread_start_us":1347,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.127.131@KRBTEST.COM" created.
I20260812 02:37:40.262319 14233 heartbeater.cc:347] Connected to a master server at 127.3.127.190:38495
I20260812 02:37:40.262593 14233 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:40.263139 14233 heartbeater.cc:511] Master 127.3.127.190:38495 requested a full tablet report, sending...
I20260812 02:37:40.264406 13692 ts_manager.cc:194] Registered new tserver with Master: 3737fe1f65fa40a484cc4f7dc3e030ba (127.3.127.130:34357)
I20260812 02:37:40.265241 13692 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:52141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.131.keytab.
Entry for principal oryx/127.3.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.131.keytab.
I20260812 02:37:40.279867 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38495
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.131.keytab
--principal=oryx/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:40.399273 14239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:40.399570 14239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:40.399614 14239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:40.403798 14239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:40.403875 14239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:40.403936 14239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:37:40.409690 14239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.131.keytab
--principal=oryx/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:38495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:40.410830 14239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:40.411855 14239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:40.421689 14248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:40.421689 14252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:40.421737 14250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:40.422191 14239 server_base.cc:1034] running on GCE node
I20260812 02:37:40.422658 14239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:40.423389 14239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:40.424582 14239 hybrid_clock.cc:648] HybridClock initialized: now 1786502260424545 us; error 48 us; skew 500 ppm
Aug 12 02:37:40 dist-test-slave-rmhg krb5kdc[13623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502260, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:40.427687 14239 init.cc:377] Logged in from keytab as oryx/127.3.127.131@KRBTEST.COM (short username oryx)
I20260812 02:37:40.428678 14239 webserver.cc:460] Webserver started at http://127.3.127.131:43769/ using document root <none> and password file <none>
I20260812 02:37:40.429311 14239 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:40.429389 14239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:40.429606 14239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:40.431703 14239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "6f0f40aab58f4476a0c404dfce26053c"
format_stamp: "Formatted at 2026-08-12 02:37:40 on dist-test-slave-rmhg"
server_key: "1d86c97aa9cefa0b7e4ae0092e6ef52b"
server_key_iv: "bbcb8fa2f6261d0cc2e76bfcab9dfdfb"
server_key_version: "encryptionkey@0"
I20260812 02:37:40.432269 14239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "6f0f40aab58f4476a0c404dfce26053c"
format_stamp: "Formatted at 2026-08-12 02:37:40 on dist-test-slave-rmhg"
server_key: "1d86c97aa9cefa0b7e4ae0092e6ef52b"
server_key_iv: "bbcb8fa2f6261d0cc2e76bfcab9dfdfb"
server_key_version: "encryptionkey@0"
I20260812 02:37:40.436215 14239 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:37:40.438827 14262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:40.440166 14239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:37:40.440284 14239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "6f0f40aab58f4476a0c404dfce26053c"
format_stamp: "Formatted at 2026-08-12 02:37:40 on dist-test-slave-rmhg"
server_key: "1d86c97aa9cefa0b7e4ae0092e6ef52b"
server_key_iv: "bbcb8fa2f6261d0cc2e76bfcab9dfdfb"
server_key_version: "encryptionkey@0"
I20260812 02:37:40.440384 14239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:40.460314 14239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:40.463707 14239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:40.463924 14239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:40.464622 14239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:40.465601 14239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:40.465654 14239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:40.465713 14239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:40.465740 14239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:40.474601 14239 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:40167
I20260812 02:37:40.474632 14447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:40167 every 8 connection(s)
I20260812 02:37:40.475610 14239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:37:40.478075 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 14239
I20260812 02:37:40.478184 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:37:40.478478 3582 external_mini_cluster.cc:1428] Setting key 37ace35083e4d0215460ca230444df01
Aug 12 02:37:40 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502260, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM
I20260812 02:37:40.490571 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:40.477639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:51567 (local address 127.3.127.190:38495)
0812 02:37:40.477824 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:40.477828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:40.479098 (+ 1270us) server_negotiation.cc:407] Connection header received
0812 02:37:40.480124 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:40.480126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:40.480175 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:40.480231 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:40.481621 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.482171 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.483449 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.483680 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.483763 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:40.486657 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:40.486661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:40.486664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:40.486696 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:40.487981 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:40.488668 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:40.488672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:40.488673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:40.488795 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:40.489230 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:40.489233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:40.489235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:40.489497 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:40.489611 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:40.490245 (+ 634us) server_negotiation.cc:299] Negotiation successful
0812 02:37:40.490396 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 02:37:40.491626 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:40.478106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.131:51567)
0812 02:37:40.478924 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:40.478986 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:40.479891 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:40.480432 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:40.480443 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:40.480882 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:40.481487 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:40.481502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.482298 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.482301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:40.483348 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:40.483359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.483853 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.483856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:40.484041 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:40.484724 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:40.484742 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:40.486424 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:40.488136 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:40.488142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:40.488158 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:40.488498 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:40.488921 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:40.488925 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:40.488927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:40.489102 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:40.489615 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:40.489623 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:40.489976 (+ 353us) client_negotiation.cc:769] Sending connection context
0812 02:37:40.490166 (+ 190us) client_negotiation.cc:240] Negotiation successful
0812 02:37:40.490574 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":138,"threads_started":1}
I20260812 02:37:40.492944 14450 heartbeater.cc:347] Connected to a master server at 127.3.127.190:38495
I20260812 02:37:40.493240 14450 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:40.493798 14450 heartbeater.cc:511] Master 127.3.127.190:38495 requested a full tablet report, sending...
I20260812 02:37:40.494973 13692 ts_manager.cc:194] Registered new tserver with Master: 6f0f40aab58f4476a0c404dfce26053c (127.3.127.131:40167)
I20260812 02:37:40.495594 13692 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:51567
I20260812 02:37:40.504967 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.127.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.127.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.127.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.189.keytab.
Entry for principal oryx/127.3.127.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.189.keytab.
I20260812 02:37:40.536409 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.189:43853
--webserver_interface=127.3.127.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.189.keytab
--principal=oryx/127.3.127.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:38495,127.3.127.189:43853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:40.655635 14459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:40.655921 14459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:40.655962 14459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:40.660046 14459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:40.660120 14459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:40.660138 14459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:40.660154 14459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:40.660167 14459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:37:40.665845 14459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:38495,127.3.127.189:43853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.189.keytab
--principal=oryx/127.3.127.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.189:43853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.127.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
W20260812 02:37:40.666095 14459 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.127.190:38495,127.3.127.189:43853'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 02:37:40.667074 14459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:40.668180 14459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:40.677464 14471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:40.677477 14475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:40.678167 14468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:40.678727 14459 server_base.cc:1034] running on GCE node
I20260812 02:37:40.679570 14459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:40.680853 14459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:40.682049 14459 hybrid_clock.cc:648] HybridClock initialized: now 1786502260682024 us; error 52 us; skew 500 ppm
Aug 12 02:37:40 dist-test-slave-rmhg krb5kdc[13623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502260, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:40.685135 14459 init.cc:377] Logged in from keytab as oryx/127.3.127.189@KRBTEST.COM (short username oryx)
I20260812 02:37:40.686133 14459 webserver.cc:460] Webserver started at http://127.3.127.189:45431/ using document root <none> and password file <none>
I20260812 02:37:40.686738 14459 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:40.686815 14459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:40.687031 14459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:40.689175 14459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/data/instance:
uuid: "dd62f6975b334750bba9d180a80ab0cd"
format_stamp: "Formatted at 2026-08-12 02:37:40 on dist-test-slave-rmhg"
server_key: "6a42a3373015c2cad1e2d7a38657f4f5"
server_key_iv: "b651bb83e6476e821fbe3daf1908e2c8"
server_key_version: "encryptionkey@0"
I20260812 02:37:40.689764 14459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/wal/instance:
uuid: "dd62f6975b334750bba9d180a80ab0cd"
format_stamp: "Formatted at 2026-08-12 02:37:40 on dist-test-slave-rmhg"
server_key: "6a42a3373015c2cad1e2d7a38657f4f5"
server_key_iv: "b651bb83e6476e821fbe3daf1908e2c8"
server_key_version: "encryptionkey@0"
I20260812 02:37:40.693617 14459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:37:40.696259 14490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:40.697716 14459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:40.697849 14459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/wal
uuid: "dd62f6975b334750bba9d180a80ab0cd"
format_stamp: "Formatted at 2026-08-12 02:37:40 on dist-test-slave-rmhg"
server_key: "6a42a3373015c2cad1e2d7a38657f4f5"
server_key_iv: "b651bb83e6476e821fbe3daf1908e2c8"
server_key_version: "encryptionkey@0"
I20260812 02:37:40.697955 14459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:40.708967 14459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:40.712534 14459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:40.712775 14459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:40.720625 14459 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.189:43853
I20260812 02:37:40.720894 14560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.189:43853 every 8 connection(s)
I20260812 02:37:40.721740 14459 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/data/info.pb
I20260812 02:37:40.724443 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 14459
I20260812 02:37:40.724833 14561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:40.730069 14561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:37:40 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502260, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.189@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM
Aug 12 02:37:40 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.127.189@KRBTEST.COM
I20260812 02:37:40.743690 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:40.732181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47012 (local address 127.3.127.190:38495)
0812 02:37:40.732298 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:40.732301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:40.732799 (+ 498us) server_negotiation.cc:407] Connection header received
0812 02:37:40.733063 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:40.733067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:40.733126 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:40.733250 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:40.733921 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.734471 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.735395 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.735660 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.735748 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:40.739615 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:40.739619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:40.739622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:40.739656 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:40.741293 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:40.741859 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:40.741863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:40.741864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:40.741961 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:40.742341 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:40.742344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:40.742346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:40.742652 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:40.742788 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:40.743372 (+ 584us) server_negotiation.cc:299] Negotiation successful
0812 02:37:40.743511 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 02:37:40.744645 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:40.732171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47012)
0812 02:37:40.732652 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:40.732710 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:40.732914 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:40.733318 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:40.733324 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:40.733669 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:40.733802 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:40.733808 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.734593 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.734595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:40.735284 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:40.735293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.735792 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.735794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:40.735878 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:40.737499 (+ 1621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:40.737518 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:40.739381 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:40.741405 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:40.741413 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:40.741428 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:40.741760 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:40.742050 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:40.742052 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:40.742055 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:40.742244 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:40.742797 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:40.742803 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:40.743121 (+ 318us) client_negotiation.cc:769] Sending connection context
0812 02:37:40.743299 (+ 178us) client_negotiation.cc:240] Negotiation successful
0812 02:37:40.743675 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":1,"thread_start_us":164,"threads_started":1}
I20260812 02:37:40.744644 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:40.726175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60166 (local address 127.3.127.189:43853)
0812 02:37:40.726619 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:40.726639 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:40.726682 (+ 43us) server_negotiation.cc:407] Connection header received
0812 02:37:40.727704 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:40.727725 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:40.728078 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:40.728376 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:40.729433 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.730496 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.731684 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.732486 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.732647 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:40.740062 (+ 7415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:40.740071 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:40.740087 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:40.740117 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:40.741816 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:40.742341 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:40.742347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:40.742349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:40.742451 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:40.742723 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:40.742727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:40.742728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:40.743129 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:40.743353 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:40.743557 (+ 204us) server_negotiation.cc:299] Negotiation successful
0812 02:37:40.743938 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":295,"spinlock_wait_cycles":896,"thread_start_us":145,"threads_started":1}
I20260812 02:37:40.746155 14561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:37:40 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502260, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.189@KRBTEST.COM for oryx/127.3.127.189@KRBTEST.COM
I20260812 02:37:40.753908 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:40.746644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:43853 (local address 127.0.0.1:60176)
0812 02:37:40.746756 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:40.746770 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:40.746845 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:40.747037 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:40.747040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:40.747183 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:40.747280 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:40.747285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.748396 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.748399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:40.748945 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:40.748953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.749390 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.749393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:40.749471 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:40.749869 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:40.749879 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:40.751073 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:40.752651 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:40.752654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:40.752655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:40.752839 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:40.753042 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:40.753043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:40.753044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:40.753109 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:40.753467 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:40.753469 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:40.753549 (+ 80us) client_negotiation.cc:769] Sending connection context
0812 02:37:40.753625 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 02:37:40.753738 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 02:37:40.753919 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:40.746720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60176 (local address 127.3.127.189:43853)
0812 02:37:40.746794 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:40.746798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:40.746810 (+ 12us) server_negotiation.cc:407] Connection header received
0812 02:37:40.746911 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:40.746914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:40.746947 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:40.747018 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:40.747366 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.748290 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.749028 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.749297 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.749370 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:40.751182 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:40.751185 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:40.751187 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:40.751208 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:40.752557 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:40.752917 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:40.752919 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:40.752920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:40.752984 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:40.753226 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:40.753228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:40.753229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:40.753397 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:40.753469 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:40.753667 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 02:37:40.753792 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":25}
I20260812 02:37:40.756858 14561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: Bootstrap starting.
I20260812 02:37:40.759359 14561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:40.760100 14561 log.cc:824] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:40.762264 14561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: No bootstrap required, opened a new log
I20260812 02:37:40.765527 14561 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } }
I20260812 02:37:40.765777 14561 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:40.765839 14561 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd62f6975b334750bba9d180a80ab0cd, State: Initialized, Role: FOLLOWER
I20260812 02:37:40.766304 14561 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } }
I20260812 02:37:40.766835 14572 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } } }
I20260812 02:37:40.766956 14572 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:37:40.767601 14561 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [sys.catalog]: configured and running, proceeding with master startup.
W20260812 02:37:40.772388 14587 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:37:40.772503 14587 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:37:40.775265 14459 master_runner.cc:428] Detected that this master dd62f6975b334750bba9d180a80ab0cd is joining an existing cluster
I20260812 02:37:40.775345 14459 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.127.189:43853
I20260812 02:37:40.786556 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:40.778101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60190 (local address 127.3.127.189:43853)
0812 02:37:40.778263 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:40.778268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:40.778625 (+ 357us) server_negotiation.cc:407] Connection header received
0812 02:37:40.778778 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:40.778780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:40.778819 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:40.778883 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:40.779415 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.780065 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.781350 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.781652 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.781746 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:40.783032 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:40.783036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:40.783038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:40.783058 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:40.784759 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:40.785326 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:40.785329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:40.785331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:40.785414 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:40.785689 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:40.785692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:40.785694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:40.785914 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:40.785996 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:40.786256 (+ 260us) server_negotiation.cc:299] Negotiation successful
0812 02:37:40.786384 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53,"spinlock_wait_cycles":896}
I20260812 02:37:40.786700 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:40.778184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47022 (local address 127.3.127.190:38495)
0812 02:37:40.778302 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:40.778306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:40.778667 (+ 361us) server_negotiation.cc:407] Connection header received
0812 02:37:40.779025 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:40.779028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:40.779083 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:40.779174 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:40.779727 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.780420 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.781289 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.781648 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.781749 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:40.783444 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:40.783449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:40.783451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:40.783484 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:40.784911 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:40.785458 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:40.785462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:40.785463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:40.785556 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:40.785837 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:40.785841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:40.785843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:40.786082 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:40.786192 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:40.786407 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 02:37:40.786536 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 02:37:40.787036 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:40.778236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:43853 (local address 127.0.0.1:60190)
0812 02:37:40.778534 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:40.778547 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:40.778686 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:40.778907 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:40.778910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:40.779179 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:40.779317 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:40.779324 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.780212 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.780216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:40.781241 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:40.781253 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.781796 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.781800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:40.781892 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:40.782425 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:40.782438 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:40.782912 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:40.784889 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:40.784893 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:40.784895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:40.785242 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:40.785496 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:40.785499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:40.785501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:40.785587 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:40.785996 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:40.785998 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:40.786091 (+ 93us) client_negotiation.cc:769] Sending connection context
0812 02:37:40.786747 (+ 656us) client_negotiation.cc:240] Negotiation successful
0812 02:37:40.786881 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":27,"spinlock_wait_cycles":15360,"thread_start_us":93,"threads_started":1}
I20260812 02:37:40.787384 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:40.778104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47022)
0812 02:37:40.778521 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:40.778536 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:40.778686 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:40.779216 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:40.779220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:40.779450 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:40.779590 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:40.779598 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.780518 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.780521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:40.781182 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:40.781193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.782199 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.782203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:40.782321 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:40.782804 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:40.782818 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:40.783297 (+ 479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:40.785020 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:40.785023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:40.785025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:40.785346 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:40.785641 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:40.785644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:40.785646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:40.785736 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:40.786179 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:40.786182 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:40.786257 (+ 75us) client_negotiation.cc:769] Sending connection context
0812 02:37:40.786351 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 02:37:40.786479 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":24,"spinlock_wait_cycles":19072,"thread_start_us":119,"threads_started":1}
Aug 12 02:37:40 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502259, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.190@KRBTEST.COM for oryx/127.3.127.189@KRBTEST.COM
I20260812 02:37:40.802062 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:40.792017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60202 (local address 127.3.127.189:43853)
0812 02:37:40.792161 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:40.792166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:40.792449 (+ 283us) server_negotiation.cc:407] Connection header received
0812 02:37:40.792678 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:40.792682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:40.792735 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:40.792810 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:40.793567 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.794076 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.795008 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.795233 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.795322 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:40.797983 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:40.797987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:40.797990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:40.798021 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:40.799868 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:40.800470 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:40.800473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:40.800475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:40.800565 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:40.800891 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:40.800896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:40.800897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:40.801237 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:40.801316 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:40.801750 (+ 434us) server_negotiation.cc:299] Negotiation successful
0812 02:37:40.801880 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 02:37:40.802269 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:40.791949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:43853 (local address 127.0.0.1:60202)
0812 02:37:40.792327 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:40.792372 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:40.792518 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:40.792869 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:40.792876 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:40.793325 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:40.793450 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:40.793457 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.794179 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.794183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:40.794889 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:40.794900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.795342 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.795346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:40.795471 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:40.796022 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:40.796036 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:40.797806 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:40.799997 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:40.800010 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:40.800018 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:40.800369 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:40.800658 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:40.800661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:40.800663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:40.800768 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:40.801397 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:40.801404 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:40.801527 (+ 123us) client_negotiation.cc:769] Sending connection context
0812 02:37:40.801663 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 02:37:40.801924 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":107,"threads_started":1}
I20260812 02:37:40.803696 13692 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.127.189:43853: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd"
member_type: NON_VOTER
last_known_addr {
host: "127.3.127.189"
port: 43853
}
attrs {
promote: true
}
}
dest_uuid: "05763676674c4a82b5e510ce078c3b07"
cas_config_opid_index: -1
I20260812 02:37:40.804422 13692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: NON_VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: true } }
I20260812 02:37:40.806339 14610 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER dd62f6975b334750bba9d180a80ab0cd (127.3.127.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: NON_VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: true } } }
I20260812 02:37:40.807365 14527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:40.808135 14610 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.127.189:43853
I20260812 02:37:40.808694 14459 master.cc:537] Master@127.3.127.189:43853 shutting down...
I20260812 02:37:40.808866 14527 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 1 FOLLOWER]: Refusing update from remote peer 05763676674c4a82b5e510ce078c3b07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 02:37:40.809602 14610 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: NON_VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:40.809701 14611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "05763676674c4a82b5e510ce078c3b07" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: NON_VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: true } } }
I20260812 02:37:40.809818 14611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [sys.catalog]: This master's current role is: LEADER
W20260812 02:37:40.812750 13676 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
WARNING: no policy specified for oryx/127.3.127.188@KRBTEST.COM; defaulting to no policy
I20260812 02:37:40.813447 14459 raft_consensus.cc:2229] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:40.813773 14459 raft_consensus.cc:2258] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 02:37:40.813784 13676 consensus_peers.cc:487] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 -> Peer dd62f6975b334750bba9d180a80ab0cd (127.3.127.189:43853): Couldn't send request to peer dd62f6975b334750bba9d180a80ab0cd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 02:37:40.813939 14459 tablet_replica.cc:330] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.127.188@KRBTEST.COM" created.
I20260812 02:37:40.828462 14459 master.cc:552] Master@127.3.127.189:43853 shutdown complete.
I20260812 02:37:40.828552 14459 master_runner.cc:305] Clearing existing system tablet
I20260812 02:37:40.828795 14459 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:40.828845 14459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:40.830231 14459 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 02:37:40.831090 14616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:40.831331 14459 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:40.831449 14459 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 02:37:40.831488 14459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/wal
uuid: "dd62f6975b334750bba9d180a80ab0cd"
format_stamp: "Formatted at 2026-08-12 02:37:40 on dist-test-slave-rmhg"
server_key: "6a42a3373015c2cad1e2d7a38657f4f5"
server_key_iv: "b651bb83e6476e821fbe3daf1908e2c8"
server_key_version: "encryptionkey@0"
I20260812 02:37:40.833750 14459 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.127.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.188.keytab.
Entry for principal oryx/127.3.127.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.188.keytab.
I20260812 02:37:40.836567 14459 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 02:37:40.836623 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.188:33229
--webserver_interface=127.3.127.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.188.keytab
--principal=oryx/127.3.127.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:38495,127.3.127.189:43853,127.3.127.188:33229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:37:40.836696 14459 log.cc:1196] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260812 02:37:40.836953 14459 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: Deleting consensus metadata
I20260812 02:37:40.837129 14459 master_runner.cc:315] Copying system tablet from 127.3.127.190:38495
I20260812 02:37:40.838644 14459 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: tablet copy: Beginning tablet copy session from remote peer at address 127.3.127.190:38495
I20260812 02:37:40.846565 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:40.839433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47030 (local address 127.3.127.190:38495)
0812 02:37:40.839570 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:40.839575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:40.839765 (+ 190us) server_negotiation.cc:407] Connection header received
0812 02:37:40.839901 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:40.839905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:40.839952 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:40.840020 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:40.840525 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.841130 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.841995 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.842227 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.842320 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:40.843433 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:40.843436 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:40.843439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:40.843466 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:40.844866 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:40.845367 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:40.845371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:40.845373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:40.845456 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:40.845718 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:40.845722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:40.845723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:40.845913 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:40.845998 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:40.846284 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 02:37:40.846399 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 02:37:40.846573 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:40.839323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47030)
0812 02:37:40.839664 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:40.839677 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:40.839802 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:40.840066 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:40.840069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:40.840277 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:40.840398 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:40.840404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.841271 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.841276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:40.841876 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:40.841885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:40.842340 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:40.842344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:40.842445 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:40.842945 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:40.842957 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:40.843289 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:40.844973 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:40.844977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:40.844979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:40.845241 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:40.845543 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:40.845547 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:40.845549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:40.845636 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:40.846003 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:40.846006 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:40.846119 (+ 113us) client_negotiation.cc:769] Sending connection context
0812 02:37:40.846207 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 02:37:40.846375 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":98,"threads_started":1}
I20260812 02:37:40.847201 13725 tablet_copy_service.cc:140] P 05763676674c4a82b5e510ce078c3b07: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer dd62f6975b334750bba9d180a80ab0cd ({username='oryx', principal='oryx/127.3.127.189@KRBTEST.COM'} at 127.0.0.1:47030)
I20260812 02:37:40.847378 13725 tablet_copy_service.cc:161] P 05763676674c4a82b5e510ce078c3b07: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer dd62f6975b334750bba9d180a80ab0cd at {username='oryx', principal='oryx/127.3.127.189@KRBTEST.COM'} at 127.0.0.1:47030: session id = dd62f6975b334750bba9d180a80ab0cd-00000000000000000000000000000000
I20260812 02:37:40.848973 13725 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 02:37:40.850780 14459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 02:37:40.852802 14459 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: tablet copy: Starting download of 0 data blocks...
I20260812 02:37:40.853128 14459 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: tablet copy: Starting download of 1 WAL segments...
I20260812 02:37:40.854846 14459 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 02:37:40.856459 13725 tablet_copy_service.cc:342] P 05763676674c4a82b5e510ce078c3b07: Request end of tablet copy session dd62f6975b334750bba9d180a80ab0cd-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.127.189@KRBTEST.COM'} at 127.0.0.1:47030
I20260812 02:37:40.856570 13725 tablet_copy_service.cc:433] P 05763676674c4a82b5e510ce078c3b07: ending tablet copy session dd62f6975b334750bba9d180a80ab0cd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer dd62f6975b334750bba9d180a80ab0cd
I20260812 02:37:40.864089 14459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:40.865883 14631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:40.865967 14636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:40.869359 14633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:40.869364 14459 server_base.cc:1034] running on GCE node
I20260812 02:37:40.869661 14459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:40.869963 14459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:40.871093 14459 hybrid_clock.cc:648] HybridClock initialized: now 1786502260871081 us; error 30 us; skew 500 ppm
Aug 12 02:37:40 dist-test-slave-rmhg krb5kdc[13623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502260, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:40.872447 14459 init.cc:377] Logged in from keytab as oryx/127.3.127.189@KRBTEST.COM (short username oryx)
I20260812 02:37:40.872882 14459 webserver.cc:460] Webserver started at http://127.3.127.189:44559/ using document root <none> and password file <none>
I20260812 02:37:40.873059 14459 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:40.873095 14459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:40.874310 14459 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 02:37:40.875187 14645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:40.875424 14459 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 02:37:40.875497 14459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/wal
uuid: "dd62f6975b334750bba9d180a80ab0cd"
format_stamp: "Formatted at 2026-08-12 02:37:40 on dist-test-slave-rmhg"
server_key: "6a42a3373015c2cad1e2d7a38657f4f5"
server_key_iv: "b651bb83e6476e821fbe3daf1908e2c8"
server_key_version: "encryptionkey@0"
I20260812 02:37:40.875584 14459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:40.884224 14459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:40.886085 14459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:40.891158 14459 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.189:43853
I20260812 02:37:40.891767 14459 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-1/data/info.pb
I20260812 02:37:40.893365 14751 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:37:40.894263 14751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: Bootstrap starting.
I20260812 02:37:40.897362 14747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.189:43853 every 8 connection(s)
I20260812 02:37:40.903443 14751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:37:40.903785 14751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: Bootstrap complete.
I20260812 02:37:40.904323 14751 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: NON_VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: true } }
I20260812 02:37:40.904454 14751 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dd62f6975b334750bba9d180a80ab0cd, State: Initialized, Role: LEARNER
I20260812 02:37:40.904583 14751 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: NON_VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: true } }
I20260812 02:37:40.904937 14754 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: NON_VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: true } } }
I20260812 02:37:40.905077 14751 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:37:40.905129 14754 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [sys.catalog]: This master's current role is: LEARNER
I20260812 02:37:40.906884 14767 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260812 02:37:40.992848 14625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:40.993115 14625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:40.993185 14625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:40.997267 14625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:40.997330 14625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:40.997349 14625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:40.997364 14625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:40.997385 14625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:37:40.998054 14005 heartbeater.cc:503] Master 127.3.127.190:38495 was elected leader, sending a full tablet report...
I20260812 02:37:41.003082 14625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:38495,127.3.127.189:43853,127.3.127.188:33229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.188.keytab
--principal=oryx/127.3.127.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.188:33229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.127.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:41.004252 14625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:41.005477 14625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:41.014174 14774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:41.014302 14625 server_base.cc:1034] running on GCE node
W20260812 02:37:41.014228 14775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:41.014187 14778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:41.015090 14625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:41.016301 14625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:41.017458 14625 hybrid_clock.cc:648] HybridClock initialized: now 1786502261017428 us; error 46 us; skew 500 ppm
Aug 12 02:37:41 dist-test-slave-rmhg krb5kdc[13623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502261, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:41.020327 14625 init.cc:377] Logged in from keytab as oryx/127.3.127.188@KRBTEST.COM (short username oryx)
I20260812 02:37:41.021421 14625 webserver.cc:460] Webserver started at http://127.3.127.188:35645/ using document root <none> and password file <none>
I20260812 02:37:41.022018 14625 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:41.022091 14625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:41.022312 14625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:41.024509 14625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/data/instance:
uuid: "0b2ff8a5d080404dbddd48b9a04d9766"
format_stamp: "Formatted at 2026-08-12 02:37:41 on dist-test-slave-rmhg"
server_key: "67baaf95e3e1831ef210acbf1d4f6de6"
server_key_iv: "b06b7ea8e6cc893da45d0ef07a7d26e4"
server_key_version: "encryptionkey@0"
I20260812 02:37:41.025101 14625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/wal/instance:
uuid: "0b2ff8a5d080404dbddd48b9a04d9766"
format_stamp: "Formatted at 2026-08-12 02:37:41 on dist-test-slave-rmhg"
server_key: "67baaf95e3e1831ef210acbf1d4f6de6"
server_key_iv: "b06b7ea8e6cc893da45d0ef07a7d26e4"
server_key_version: "encryptionkey@0"
I20260812 02:37:41.029217 14625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:37:41.031819 14790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:41.033195 14625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:41.033316 14625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/wal
uuid: "0b2ff8a5d080404dbddd48b9a04d9766"
format_stamp: "Formatted at 2026-08-12 02:37:41 on dist-test-slave-rmhg"
server_key: "67baaf95e3e1831ef210acbf1d4f6de6"
server_key_iv: "b06b7ea8e6cc893da45d0ef07a7d26e4"
server_key_version: "encryptionkey@0"
I20260812 02:37:41.033420 14625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:41.049028 14625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:41.052690 14625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:41.052901 14625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:41.061031 14625 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.188:33229
I20260812 02:37:41.061342 14876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.188:33229 every 8 connection(s)
I20260812 02:37:41.062165 14625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/data/info.pb
I20260812 02:37:41.065219 14877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:41.066553 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 14625
I20260812 02:37:41.069859 14877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:37:41 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.127.188@KRBTEST.COM
Aug 12 02:37:41 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502261, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.188@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM
I20260812 02:37:41.081625 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.071848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47036 (local address 127.3.127.190:38495)
0812 02:37:41.071995 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.071999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.072277 (+ 278us) server_negotiation.cc:407] Connection header received
0812 02:37:41.072402 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.072405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.072454 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.072548 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.073452 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.074305 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.075482 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.075781 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.075877 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.078515 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.078518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.078520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.078549 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.079844 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.080303 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.080307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.080309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.080390 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.080642 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.080645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.080647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.080834 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.080922 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.081350 (+ 428us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.081479 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 02:37:41.082566 14879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.068396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52460 (local address 127.3.127.188:33229)
0812 02:37:41.068912 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.068929 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.068973 (+ 44us) server_negotiation.cc:407] Connection header received
0812 02:37:41.069720 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.069740 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.070118 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.070479 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.071343 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.072657 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.073815 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.074568 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.074697 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.077740 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.077746 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.077758 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.077783 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.079443 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.079932 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.079938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.079943 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.080059 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.080332 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.080336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.080337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.080788 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.081010 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.081259 (+ 249us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.081600 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":170,"threads_started":1}
I20260812 02:37:41.082564 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.071593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47036)
0812 02:37:41.072144 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.072194 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.072318 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.072624 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.072630 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.073091 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:41.073294 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.073302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.074462 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.074464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.075348 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.075362 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.075960 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.075965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.076094 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.076775 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:41.076800 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:41.078340 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:41.079956 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.079966 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.079973 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.080204 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.080473 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.080476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.080478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.080558 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.080933 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:41.080941 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:41.081079 (+ 138us) client_negotiation.cc:769] Sending connection context
0812 02:37:41.081267 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 02:37:41.081713 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":106,"thread_start_us":119,"threads_started":1}
I20260812 02:37:41.084036 14877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:37:41 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502261, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.188@KRBTEST.COM for oryx/127.3.127.189@KRBTEST.COM
I20260812 02:37:41.092242 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.084554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:43853 (local address 127.0.0.1:60204)
0812 02:37:41.084738 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.084751 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.084830 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.085222 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.085224 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.085369 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:41.085474 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.085479 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.086328 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.086331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.086860 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.086868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.087368 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.087371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.087468 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.088136 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:41.088148 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:41.089412 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:41.091069 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.091072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.091073 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.091261 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.091475 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.091477 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.091478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.091543 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.091842 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:41.091844 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:41.091930 (+ 86us) client_negotiation.cc:769] Sending connection context
0812 02:37:41.092002 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 02:37:41.092115 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260812 02:37:41.092294 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.084690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.3.127.189:43853)
0812 02:37:41.084966 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.084969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.084982 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:37:41.085020 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.085022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.085065 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.085181 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.085570 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.086232 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.086979 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.087233 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.087316 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.089562 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.089566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.089568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.089594 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.090962 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.091336 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.091339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.091340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.091414 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.091599 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.091601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.091602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.091781 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.091850 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.092046 (+ 196us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.092150 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":114,"threads_started":1}
I20260812 02:37:41.093056 14877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:37:41 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502261, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.188@KRBTEST.COM for oryx/127.3.127.188@KRBTEST.COM
I20260812 02:37:41.101372 14879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.093678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.3.127.188:33229)
0812 02:37:41.093812 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.093815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.093828 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:37:41.093885 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.093887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.093922 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.093993 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.094324 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.094834 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.095529 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.095792 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.095874 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.097865 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.097869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.097871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.097894 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.099778 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.100190 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.100194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.100195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.100277 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.100476 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.100478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.100479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.100668 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.100789 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.101051 (+ 262us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.101236 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":59}
I20260812 02:37:41.101372 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.093608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:33229 (local address 127.0.0.1:52462)
0812 02:37:41.093736 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.093749 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.093819 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.094002 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.094004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.094133 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:41.094214 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.094218 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.094909 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.094911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.095417 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.095423 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.095907 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.095910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.095977 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.096442 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:41.096454 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:41.097718 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:41.099887 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.099890 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.099892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.100085 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.100349 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.100351 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.100352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.100417 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.100802 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:41.100806 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:41.100893 (+ 87us) client_negotiation.cc:769] Sending connection context
0812 02:37:41.100994 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 02:37:41.101179 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260812 02:37:41.104658 14877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766: Bootstrap starting.
I20260812 02:37:41.107300 14877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:41.108045 14877 log.cc:824] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:41.110080 14877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766: No bootstrap required, opened a new log
I20260812 02:37:41.113364 14877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } }
I20260812 02:37:41.113651 14877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:41.113699 14877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b2ff8a5d080404dbddd48b9a04d9766, State: Initialized, Role: FOLLOWER
I20260812 02:37:41.114126 14877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } }
I20260812 02:37:41.114617 14885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } } }
I20260812 02:37:41.114759 14885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:37:41.115391 14877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 02:37:41.120580 14900 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:37:41.120694 14900 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:37:41.124131 14625 master_runner.cc:428] Detected that this master 0b2ff8a5d080404dbddd48b9a04d9766 is joining an existing cluster
I20260812 02:37:41.124202 14625 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.127.188:33229
I20260812 02:37:41.135581 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.126558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47048 (local address 127.3.127.190:38495)
0812 02:37:41.126691 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.126695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.127454 (+ 759us) server_negotiation.cc:407] Connection header received
0812 02:37:41.127623 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.127626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.127690 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.127743 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.128353 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.129109 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.130362 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.130646 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.130748 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.132031 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.132034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.132036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.132063 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.133526 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.134075 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.134079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.134080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.134188 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.134543 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.134546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.134547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.134780 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.134871 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.135295 (+ 424us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.135429 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 02:37:41.135962 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.126719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47048)
0812 02:37:41.127371 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.127383 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.127519 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.127879 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.127882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.128100 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:41.128220 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.128227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.129288 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.129292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.130245 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.130257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.130858 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.130862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.130963 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.131387 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:41.131401 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:41.131866 (+ 465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:41.133672 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.133675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.133678 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.133961 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.134286 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.134290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.134292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.134401 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.134878 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:41.134882 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:41.135087 (+ 205us) client_negotiation.cc:769] Sending connection context
0812 02:37:41.135225 (+ 138us) client_negotiation.cc:240] Negotiation successful
0812 02:37:41.135387 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":96,"threads_started":1}
I20260812 02:37:41.136469 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 13794
Aug 12 02:37:41 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502259, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.190@KRBTEST.COM for oryx/127.3.127.188@KRBTEST.COM
I20260812 02:37:41.179102 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.150755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:33229 (local address 127.0.0.1:52480)
0812 02:37:41.151046 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.151059 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.151165 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.153534 (+ 2369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.153538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.153786 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:41.153930 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.153938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.154989 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.154993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.155671 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.155683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.156367 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.156371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.156514 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.157029 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:41.157042 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:41.158586 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:41.176635 (+ 18049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.176640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.176642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.176961 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.177366 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.177370 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.177372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.177474 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.178252 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:41.178255 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:41.178370 (+ 115us) client_negotiation.cc:769] Sending connection context
0812 02:37:41.178709 (+ 339us) client_negotiation.cc:240] Negotiation successful
0812 02:37:41.178902 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260812 02:37:41.180274 14917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.151592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52480 (local address 127.3.127.188:33229)
0812 02:37:41.153038 (+ 1446us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.153056 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.153077 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:37:41.153243 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.153246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.153324 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.154056 (+ 732us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.154204 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.154867 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.155815 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.156236 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.158786 (+ 2550us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.158949 (+ 163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.158951 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.158953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.158967 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.176459 (+ 17492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.177112 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.177116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.177118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.177248 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.177869 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.177872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.177874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.178132 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.179776 (+ 1644us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.179861 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.179996 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1353,"thread_start_us":1183,"threads_started":1}
I20260812 02:37:41.181401 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:37785
--local_ip_for_outbound_sockets=127.3.127.129
--tserver_master_addrs=127.3.127.190:38495
--webserver_port=39403
--webserver_interface=127.3.127.129
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.129.keytab
--principal=oryx/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.127.190:38495,127.3.127.189:43853
--tserver_master_addrs=127.3.127.190:38495,127.3.127.189:43853,127.3.127.188:33229 with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:37:41.181820 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.126547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60210 (local address 127.3.127.189:43853)
0812 02:37:41.126685 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.126689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.126924 (+ 235us) server_negotiation.cc:407] Connection header received
0812 02:37:41.127043 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.127047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.127098 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.127179 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.127836 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.128580 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.129850 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.130087 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.130178 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.177217 (+ 47039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.177222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.177224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.177261 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.179359 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.180383 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.180388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.180389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.180496 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.180804 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.180807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.180809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.181043 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.181133 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.181535 (+ 402us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.181681 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 02:37:41.182152 13692 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.127.188:33229: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766"
member_type: NON_VOTER
last_known_addr {
host: "127.3.127.188"
port: 33229
}
attrs {
promote: true
}
}
dest_uuid: "05763676674c4a82b5e510ce078c3b07"
cas_config_opid_index: 5
I20260812 02:37:41.182444 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.126552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:43853 (local address 127.0.0.1:60210)
0812 02:37:41.126847 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.126861 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.126966 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.127313 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.127316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.127563 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:41.127707 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.127715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.128725 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.128728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.129733 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.129742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.130207 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.130211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.130309 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.130839 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:41.130853 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:41.176922 (+ 46069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:41.179490 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.179493 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.179495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.179744 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.180612 (+ 868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.180616 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.180617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.180705 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.181236 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:41.181238 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:41.181339 (+ 101us) client_negotiation.cc:769] Sending connection context
0812 02:37:41.182015 (+ 676us) client_negotiation.cc:240] Negotiation successful
0812 02:37:41.182178 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":115,"threads_started":1}
I20260812 02:37:41.182861 13692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: NON_VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: true } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: NON_VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: true } }
I20260812 02:37:41.185088 14611 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0b2ff8a5d080404dbddd48b9a04d9766 (127.3.127.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: NON_VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: true } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: NON_VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: true } } }
I20260812 02:37:41.185590 14837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:41.187101 14611 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.127.188:33229
I20260812 02:37:41.187326 14611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "05763676674c4a82b5e510ce078c3b07" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: NON_VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: true } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: NON_VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: true } } }
I20260812 02:37:41.187423 14611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:41.188628 14837 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 1 FOLLOWER]: Refusing update from remote peer 05763676674c4a82b5e510ce078c3b07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 02:37:41.189337 14625 master.cc:537] Master@127.3.127.188:33229 shutting down...
I20260812 02:37:41.191293 14610 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: NON_VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:41.191632 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.127135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:33229 (local address 127.0.0.1:52474)
0812 02:37:41.127414 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.127427 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.127558 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.127845 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.127848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.173853 (+ 46005us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:41.174254 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.174272 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.175753 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.175757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.176628 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.176644 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.178560 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.178563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.179905 (+ 1342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.183485 (+ 3580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:41.183500 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:41.183954 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:41.189835 (+ 5881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.189839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.189842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.190214 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.190517 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.190519 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.190521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.190602 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.191061 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:41.191063 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:41.191170 (+ 107us) client_negotiation.cc:769] Sending connection context
0812 02:37:41.191282 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 02:37:41.191442 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":81,"threads_started":1}
I20260812 02:37:41.192129 14879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.127114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52474 (local address 127.3.127.188:33229)
0812 02:37:41.127219 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.127223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.127489 (+ 266us) server_negotiation.cc:407] Connection header received
0812 02:37:41.127647 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.127650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.127687 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.127728 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.174438 (+ 46710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.175229 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.177616 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.177957 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.178047 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.184094 (+ 6047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.184097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.184100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.184124 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.189456 (+ 5332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.190335 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.190338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.190340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.190424 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.190693 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.190696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.190697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.190931 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.191027 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.191855 (+ 828us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.191973 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":47}
W20260812 02:37:41.193719 13674 consensus_peers.cc:487] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 -> Peer 0b2ff8a5d080404dbddd48b9a04d9766 (127.3.127.188:33229): Couldn't send request to peer 0b2ff8a5d080404dbddd48b9a04d9766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 02:37:41.197322 14625 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:41.197755 14625 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:41.197914 14625 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766: stopping tablet replica
I20260812 02:37:41.204542 14625 master.cc:552] Master@127.3.127.188:33229 shutdown complete.
I20260812 02:37:41.204612 14625 master_runner.cc:305] Clearing existing system tablet
I20260812 02:37:41.204879 14625 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:41.204943 14625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:41.205276 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.204795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:33229 (local address 127.0.0.1:52490)
0812 02:37:41.205086 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.205178 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:33229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260812 02:37:41.206660 14625 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 02:37:41.206753 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.184861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:43853 (local address 127.0.0.1:60212)
0812 02:37:41.194015 (+ 9154us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.194033 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.194149 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.194551 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.194555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.194763 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:41.194882 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.194889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.197056 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.197060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.197985 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.198001 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.198780 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.198784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.198883 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.199631 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:41.199647 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:41.200807 (+ 1160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:41.203717 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.203720 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.203723 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.204015 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.204329 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.204332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.204334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.204420 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.206191 (+ 1771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:41.206195 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:41.206315 (+ 120us) client_negotiation.cc:769] Sending connection context
0812 02:37:41.206405 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 02:37:41.206578 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9055}
I20260812 02:37:41.207356 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.184731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60212 (local address 127.3.127.189:43853)
0812 02:37:41.189314 (+ 4583us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.189320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.194107 (+ 4787us) server_negotiation.cc:407] Connection header received
0812 02:37:41.194265 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.194269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.194334 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.194415 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.195944 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.196870 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.198153 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.198521 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.198622 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.201598 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.201603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.201606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.201644 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.203578 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.204134 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.204138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.204140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.204233 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.205723 (+ 1490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.205727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.205729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.206054 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.206939 (+ 885us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.207061 (+ 122us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.207199 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4465}
I20260812 02:37:41.207548 14920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:41.207736 14625 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:41.207864 14625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 02:37:41.207862 14699 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 1 LEARNER]: Refusing update from remote peer 05763676674c4a82b5e510ce078c3b07: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 02:37:41.207899 14625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/wal
uuid: "0b2ff8a5d080404dbddd48b9a04d9766"
format_stamp: "Formatted at 2026-08-12 02:37:41 on dist-test-slave-rmhg"
server_key: "67baaf95e3e1831ef210acbf1d4f6de6"
server_key_iv: "b06b7ea8e6cc893da45d0ef07a7d26e4"
server_key_version: "encryptionkey@0"
I20260812 02:37:41.208361 14610 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: NON_VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 02:37:41.210490 14625 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:41.210687 14699 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0b2ff8a5d080404dbddd48b9a04d9766 (127.3.127.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: NON_VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: true } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: NON_VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: true } } }
I20260812 02:37:41.213006 14925 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05763676674c4a82b5e510ce078c3b07. Latest consensus state: current_term: 1 leader_uuid: "05763676674c4a82b5e510ce078c3b07" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: NON_VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: true } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: NON_VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: true } } }
I20260812 02:37:41.213189 14925 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [sys.catalog]: This master's current role is: LEARNER
I20260812 02:37:41.213259 14754 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05763676674c4a82b5e510ce078c3b07" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: NON_VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: true } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: NON_VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: true } } }
I20260812 02:37:41.213385 14754 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [sys.catalog]: This master's current role is: LEARNER
I20260812 02:37:41.213137 14625 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 02:37:41.214123 14625 log.cc:1196] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260812 02:37:41.214411 14625 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766: Deleting consensus metadata
I20260812 02:37:41.214584 14625 master_runner.cc:315] Copying system tablet from 127.3.127.190:38495
I20260812 02:37:41.214802 14610 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07: attempting to promote NON_VOTER dd62f6975b334750bba9d180a80ab0cd to VOTER
I20260812 02:37:41.215399 14610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: NON_VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: true } }
I20260812 02:37:41.216179 14625 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766: tablet copy: Beginning tablet copy session from remote peer at address 127.3.127.190:38495
I20260812 02:37:41.217504 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.216742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:33229 (local address 127.0.0.1:52494)
0812 02:37:41.217311 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.217409 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:33229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480}
I20260812 02:37:41.218175 14699 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 1 LEARNER]: Refusing update from remote peer 05763676674c4a82b5e510ce078c3b07: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 02:37:41.219039 14611 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 02:37:41.220136 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.219849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:33229 (local address 127.0.0.1:52504)
0812 02:37:41.219994 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.220049 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:33229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260812 02:37:41.220903 13674 consensus_peers.cc:487] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 -> Peer 0b2ff8a5d080404dbddd48b9a04d9766 (127.3.127.188:33229): Couldn't send request to peer 0b2ff8a5d080404dbddd48b9a04d9766. Status: Network error: Client connection negotiation failed: client connection to 127.3.127.188:33229: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 02:37:41.222127 14611 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, dd62f6975b334750bba9d180a80ab0cd (127.3.127.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: NON_VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: true } } }
I20260812 02:37:41.224243 14699 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, dd62f6975b334750bba9d180a80ab0cd (127.3.127.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: NON_VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: true } } }
I20260812 02:37:41.226050 14611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "05763676674c4a82b5e510ce078c3b07" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: NON_VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: true } } }
I20260812 02:37:41.226181 14611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:41.226143 14754 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05763676674c4a82b5e510ce078c3b07" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: NON_VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: true } } }
I20260812 02:37:41.226284 14754 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:37:41.226307 14611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05763676674c4a82b5e510ce078c3b07" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: NON_VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: true } } }
I20260812 02:37:41.226375 14611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:41.228673 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.217184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47056 (local address 127.3.127.190:38495)
0812 02:37:41.217310 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.217314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.217885 (+ 571us) server_negotiation.cc:407] Connection header received
0812 02:37:41.218081 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.218085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.218136 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.218188 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.218914 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.219736 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.221486 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.221828 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.221912 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.223486 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.223491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.223494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.223524 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.225346 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.227262 (+ 1916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.227266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.227267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.227372 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.227703 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.227706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.227709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.227966 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.228062 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.228371 (+ 309us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.228506 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 02:37:41.228705 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.217421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47056)
0812 02:37:41.217792 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.217806 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.217974 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.218371 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.218374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.218607 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:41.218770 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.218778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.219977 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.219980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.221075 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.221089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.222096 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.222099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.222203 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.222860 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:41.222877 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:41.223324 (+ 447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:41.226836 (+ 3512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.226841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.226843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.227147 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.227488 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.227491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.227493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.227602 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.228062 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:41.228066 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:41.228195 (+ 129us) client_negotiation.cc:769] Sending connection context
0812 02:37:41.228309 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 02:37:41.228482 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":79,"threads_started":1}
I20260812 02:37:41.229252 13725 tablet_copy_service.cc:140] P 05763676674c4a82b5e510ce078c3b07: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0b2ff8a5d080404dbddd48b9a04d9766 ({username='oryx', principal='oryx/127.3.127.188@KRBTEST.COM'} at 127.0.0.1:47056)
I20260812 02:37:41.229375 13725 tablet_copy_service.cc:161] P 05763676674c4a82b5e510ce078c3b07: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0b2ff8a5d080404dbddd48b9a04d9766 at {username='oryx', principal='oryx/127.3.127.188@KRBTEST.COM'} at 127.0.0.1:47056: session id = 0b2ff8a5d080404dbddd48b9a04d9766-00000000000000000000000000000000
I20260812 02:37:41.230476 13725 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 02:37:41.231966 14625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 02:37:41.235198 14625 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766: tablet copy: Starting download of 0 data blocks...
I20260812 02:37:41.235430 14625 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766: tablet copy: Starting download of 1 WAL segments...
I20260812 02:37:41.240907 14625 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 02:37:41.243425 13725 tablet_copy_service.cc:342] P 05763676674c4a82b5e510ce078c3b07: Request end of tablet copy session 0b2ff8a5d080404dbddd48b9a04d9766-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.127.188@KRBTEST.COM'} at 127.0.0.1:47056
I20260812 02:37:41.243551 13725 tablet_copy_service.cc:433] P 05763676674c4a82b5e510ce078c3b07: ending tablet copy session 0b2ff8a5d080404dbddd48b9a04d9766-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0b2ff8a5d080404dbddd48b9a04d9766
I20260812 02:37:41.252534 14625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:41.254498 14939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:41.254761 14936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:41.255013 14625 server_base.cc:1034] running on GCE node
W20260812 02:37:41.257793 14935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:41.258090 14625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:41.258457 14625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:41.259611 14625 hybrid_clock.cc:648] HybridClock initialized: now 1786502261259591 us; error 39 us; skew 500 ppm
Aug 12 02:37:41 dist-test-slave-rmhg krb5kdc[13623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502261, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:41.261385 14625 init.cc:377] Logged in from keytab as oryx/127.3.127.188@KRBTEST.COM (short username oryx)
I20260812 02:37:41.261934 14625 webserver.cc:460] Webserver started at http://127.3.127.188:45409/ using document root <none> and password file <none>
I20260812 02:37:41.262156 14625 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:41.262199 14625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:41.263751 14625 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 02:37:41.264866 14951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:41.265239 14625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 02:37:41.265326 14625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/wal
uuid: "0b2ff8a5d080404dbddd48b9a04d9766"
format_stamp: "Formatted at 2026-08-12 02:37:41 on dist-test-slave-rmhg"
server_key: "67baaf95e3e1831ef210acbf1d4f6de6"
server_key_iv: "b06b7ea8e6cc893da45d0ef07a7d26e4"
server_key_version: "encryptionkey@0"
I20260812 02:37:41.265419 14625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:41.268165 14233 heartbeater.cc:503] Master 127.3.127.190:38495 was elected leader, sending a full tablet report...
I20260812 02:37:41.278910 14625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:41.281443 14625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:41.287401 14625 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.188:33229
I20260812 02:37:41.287432 15045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.188:33229 every 8 connection(s)
I20260812 02:37:41.288084 14625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/master-2/data/info.pb
I20260812 02:37:41.290321 15046 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:37:41.291626 15046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766: Bootstrap starting.
I20260812 02:37:41.305830 15046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:37:41.306248 15046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766: Bootstrap complete.
I20260812 02:37:41.307049 15046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: NON_VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: true } }
I20260812 02:37:41.307235 15046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0b2ff8a5d080404dbddd48b9a04d9766, State: Initialized, Role: LEARNER
I20260812 02:37:41.307351 15046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: NON_VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: true } }
I20260812 02:37:41.307775 15051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: NON_VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: true } } }
I20260812 02:37:41.307883 15046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:37:41.308012 15051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [sys.catalog]: This master's current role is: LEARNER
I20260812 02:37:41.312220 15067 catalog_manager.cc:1259] Loaded cluster ID: 1edca9d490854f83b1c360530e4749c0
I20260812 02:37:41.312397 15067 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766: loading cluster ID for follower catalog manager: success
I20260812 02:37:41.314811 15067 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766: acquiring CA information for follower catalog manager: success
I20260812 02:37:41.316483 15067 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 02:37:41.358369 14918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:41.358755 14918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:41.358830 14918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:41.363476 14918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:41.363564 14918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:41.363649 14918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:37:41.370187 14918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.129.keytab
--principal=oryx/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:37785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=39403
--tserver_master_addrs=127.3.127.190:38495,127.3.127.189:43853,127.3.127.188:33229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:41.371654 14918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:41.372826 14918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:41.385447 15077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:41.385474 15075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:41.385447 15074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:41.385537 14918 server_base.cc:1034] running on GCE node
I20260812 02:37:41.386178 14918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:41.386859 14918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:41.388047 14918 hybrid_clock.cc:648] HybridClock initialized: now 1786502261388015 us; error 46 us; skew 500 ppm
Aug 12 02:37:41 dist-test-slave-rmhg krb5kdc[13623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502261, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:41.391647 14918 init.cc:377] Logged in from keytab as oryx/127.3.127.129@KRBTEST.COM (short username oryx)
I20260812 02:37:41.392637 14918 webserver.cc:460] Webserver started at http://127.3.127.129:39403/ using document root <none> and password file <none>
I20260812 02:37:41.393294 14918 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:41.393384 14918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:41.398286 14918 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:37:41.400614 15086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:41.402180 14918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 02:37:41.402308 14918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "430e867652ed41ff8cda09459a710434"
format_stamp: "Formatted at 2026-08-12 02:37:39 on dist-test-slave-rmhg"
server_key: "7323242eca7db23784291bf347ff5d91"
server_key_iv: "e8e27b3635d754102e77e68e7b1fcfd6"
server_key_version: "encryptionkey@0"
I20260812 02:37:41.403079 14918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:41.420671 14918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:41.424836 14918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:41.425097 14918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:41.426009 14918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:41.427079 14918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:41.427130 14918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:41.427174 14918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:41.427196 14918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:41.448727 14918 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:37785
I20260812 02:37:41.448760 15261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:37785 every 8 connection(s)
I20260812 02:37:41.449975 14918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:37:41.454257 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 14918
I20260812 02:37:41.454466 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 14013
Aug 12 02:37:41 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502261, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM
Aug 12 02:37:41 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502261, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.188@KRBTEST.COM
Aug 12 02:37:41 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502261, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.189@KRBTEST.COM
I20260812 02:37:41.480014 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:34357
--local_ip_for_outbound_sockets=127.3.127.130
--tserver_master_addrs=127.3.127.190:38495
--webserver_port=46649
--webserver_interface=127.3.127.130
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.130.keytab
--principal=oryx/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.127.190:38495,127.3.127.189:43853
--tserver_master_addrs=127.3.127.190:38495,127.3.127.189:43853,127.3.127.188:33229 with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:37:41.480661 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.452084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:47329 (local address 127.3.127.190:38495)
0812 02:37:41.452228 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.452233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.453578 (+ 1345us) server_negotiation.cc:407] Connection header received
0812 02:37:41.455102 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.455106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.455163 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.455223 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.457845 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.458907 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.461723 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.462153 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.462251 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.471917 (+ 9666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.471921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.471924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.471959 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.475727 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.476985 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.476990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.476992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.477107 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.478361 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.478366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.478368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.478673 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.478774 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.480340 (+ 1566us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.480499 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 02:37:41.482554 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.452589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.129:47329)
0812 02:37:41.453409 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.453465 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.454819 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.455463 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.455476 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.456356 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:41.457596 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.457639 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.459104 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.459108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.461521 (+ 2413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.461537 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.462536 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.462541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.462901 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.469041 (+ 6140us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:41.469065 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:41.471656 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:41.476006 (+ 4350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.476040 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.476064 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.476817 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.477930 (+ 1113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.477933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.477935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.478179 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.478860 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:41.478867 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:41.479257 (+ 390us) client_negotiation.cc:769] Sending connection context
0812 02:37:41.480848 (+ 1591us) client_negotiation.cc:240] Negotiation successful
0812 02:37:41.481366 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"spinlock_wait_cycles":33152,"thread_start_us":297,"threads_started":1}
I20260812 02:37:41.485057 15264 heartbeater.cc:347] Connected to a master server at 127.3.127.190:38495
I20260812 02:37:41.485386 15264 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:41.486070 15264 heartbeater.cc:511] Master 127.3.127.190:38495 requested a full tablet report, sending...
I20260812 02:37:41.487721 13692 ts_manager.cc:194] Re-registered known tserver with Master: 430e867652ed41ff8cda09459a710434 (127.3.127.129:37785)
I20260812 02:37:41.487855 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.452103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:33997 (local address 127.3.127.189:43853)
0812 02:37:41.452236 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.452241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.453580 (+ 1339us) server_negotiation.cc:407] Connection header received
0812 02:37:41.463666 (+ 10086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.463671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.463751 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.463818 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.465181 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.465945 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.467672 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.468013 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.468098 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.483657 (+ 15559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.483662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.483664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.483700 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.485419 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.485976 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.485980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.485981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.486092 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.486397 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.486401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.486403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.486671 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.486772 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.487573 (+ 801us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.487707 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 02:37:41.487954 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.454040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:54261 (local address 127.3.127.188:33229)
0812 02:37:41.454308 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.454311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.454692 (+ 381us) server_negotiation.cc:407] Connection header received
0812 02:37:41.465121 (+ 10429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.465126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.465218 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.465359 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.466444 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.467548 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.469462 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.469911 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.470031 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.479674 (+ 9643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.479679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.479682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.479716 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.483788 (+ 4072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.484721 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.484726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.484728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.484830 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.486905 (+ 2075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.486909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.486911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.487189 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.487283 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.487656 (+ 373us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.487802 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":104,"threads_started":1}
I20260812 02:37:41.488371 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.452589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:43853 (local address 127.3.127.129:33997)
0812 02:37:41.453409 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.453465 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.463503 (+ 10038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.463997 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.464000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.464525 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:41.464998 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.465010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.466138 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.466142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.467513 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.467527 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.468276 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.468279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.468606 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.470906 (+ 2300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:41.470920 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:41.479816 (+ 8896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:41.485578 (+ 5762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.485581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.485583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.485842 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.486216 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.486219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.486221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.486311 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.487248 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:41.487251 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:41.487357 (+ 106us) client_negotiation.cc:769] Sending connection context
0812 02:37:41.488042 (+ 685us) client_negotiation.cc:240] Negotiation successful
0812 02:37:41.488200 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":24,"spinlock_wait_cycles":20608,"thread_start_us":301,"threads_started":1}
I20260812 02:37:41.488513 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.453888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:33229 (local address 127.3.127.129:54261)
0812 02:37:41.454605 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.454617 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.463389 (+ 8772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.465500 (+ 2111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.465503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.465854 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:41.466237 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.466245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.467666 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.467669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.469299 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.469313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.470355 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.470363 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.470535 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.471629 (+ 1094us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:41.471644 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:41.474157 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:41.484277 (+ 10120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.484281 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.484284 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.484579 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.484939 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.484942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.484944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.486773 (+ 1829us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.487349 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:41.487352 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:41.487450 (+ 98us) client_negotiation.cc:769] Sending connection context
0812 02:37:41.488157 (+ 707us) client_negotiation.cc:240] Negotiation successful
0812 02:37:41.488312 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":659,"thread_start_us":172,"threads_started":1}
I20260812 02:37:41.488507 13692 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:47329
I20260812 02:37:41.489106 15267 heartbeater.cc:347] Connected to a master server at 127.3.127.189:43853
I20260812 02:37:41.489234 15267 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:41.489450 15267 heartbeater.cc:511] Master 127.3.127.189:43853 requested a full tablet report, sending...
I20260812 02:37:41.489894 15263 heartbeater.cc:347] Connected to a master server at 127.3.127.188:33229
I20260812 02:37:41.489997 15263 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:41.490165 15263 heartbeater.cc:511] Master 127.3.127.188:33229 requested a full tablet report, sending...
I20260812 02:37:41.490770 14684 ts_manager.cc:194] Registered new tserver with Master: 430e867652ed41ff8cda09459a710434 (127.3.127.129:37785)
I20260812 02:37:41.492533 14974 ts_manager.cc:194] Registered new tserver with Master: 430e867652ed41ff8cda09459a710434 (127.3.127.129:37785)
I20260812 02:37:41.497797 14450 heartbeater.cc:503] Master 127.3.127.190:38495 was elected leader, sending a full tablet report...
W20260812 02:37:41.607452 15274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:41.607852 15274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:41.607929 15274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:41.613205 15274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:41.613278 15274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:41.613364 15274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:37:41.620870 15274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.130.keytab
--principal=oryx/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:34357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=46649
--tserver_master_addrs=127.3.127.190:38495,127.3.127.189:43853,127.3.127.188:33229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:41.622490 15274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:41.623867 15274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:41.634603 15284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:41.634677 15274 server_base.cc:1034] running on GCE node
W20260812 02:37:41.634604 15289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:41.634966 15285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:41.635435 15274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:41.636157 15274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:41.637322 15274 hybrid_clock.cc:648] HybridClock initialized: now 1786502261637299 us; error 37 us; skew 500 ppm
Aug 12 02:37:41 dist-test-slave-rmhg krb5kdc[13623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502261, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:41.640334 15274 init.cc:377] Logged in from keytab as oryx/127.3.127.130@KRBTEST.COM (short username oryx)
I20260812 02:37:41.641433 15274 webserver.cc:460] Webserver started at http://127.3.127.130:46649/ using document root <none> and password file <none>
I20260812 02:37:41.642060 15274 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:41.642143 15274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:41.646873 15274 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:37:41.649363 15299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:41.650804 15274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:41.650933 15274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "3737fe1f65fa40a484cc4f7dc3e030ba"
format_stamp: "Formatted at 2026-08-12 02:37:40 on dist-test-slave-rmhg"
server_key: "25ed4d997c779efd189a5798b1216215"
server_key_iv: "4430d0037ca540e5a4e364420ec846ec"
server_key_version: "encryptionkey@0"
I20260812 02:37:41.651528 15274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:41.702442 15274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:41.706498 15274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:41.706727 15274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:41.707433 15274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:41.708454 15274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:41.708505 15274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:41.708555 15274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:41.708587 15274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:41.720561 15274 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:34357
I20260812 02:37:41.720690 15476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:34357 every 8 connection(s)
I20260812 02:37:41.721876 15274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:37:41.723193 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 15274
I20260812 02:37:41.723361 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 14239
Aug 12 02:37:41 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502261, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM
Aug 12 02:37:41 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502261, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.189@KRBTEST.COM
Aug 12 02:37:41 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502261, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.188@KRBTEST.COM
I20260812 02:37:41.748545 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:40167
--local_ip_for_outbound_sockets=127.3.127.131
--tserver_master_addrs=127.3.127.190:38495
--webserver_port=43769
--webserver_interface=127.3.127.131
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.131.keytab
--principal=oryx/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.127.190:38495,127.3.127.189:43853
--tserver_master_addrs=127.3.127.190:38495,127.3.127.189:43853,127.3.127.188:33229 with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:37:41.753420 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.739332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:52497 (local address 127.3.127.190:38495)
0812 02:37:41.739456 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.739460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.739751 (+ 291us) server_negotiation.cc:407] Connection header received
0812 02:37:41.739922 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.739925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.739974 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.740024 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.740708 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.741585 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.743781 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.744148 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.744224 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.748597 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.748602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.748605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.748638 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.750569 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.751339 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.751343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.751345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.751446 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.751905 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.751908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.751910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.752153 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.752236 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.753068 (+ 832us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.753242 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 02:37:41.753420 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.737626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:41511 (local address 127.3.127.188:33229)
0812 02:37:41.737789 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.737794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.737973 (+ 179us) server_negotiation.cc:407] Connection header received
0812 02:37:41.738108 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.738111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.738170 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.738247 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.740712 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.741593 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.743782 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.744145 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.744224 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.748710 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.748714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.748716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.748751 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.750808 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.751404 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.751409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.751410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.751498 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.751905 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.751908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.751910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.752181 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.752244 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.753068 (+ 824us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.753246 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 02:37:41.754724 15485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.724606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:43853 (local address 127.3.127.130:52747)
0812 02:37:41.733421 (+ 8815us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.733507 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.735116 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.740869 (+ 5753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.740872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.741038 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:41.741186 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.741194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.742263 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.742266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.743648 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.743663 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.744451 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.744455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.744733 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.745565 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:41.745612 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:41.748340 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:41.750772 (+ 2432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.750775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.750778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.751218 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.751549 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.751552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.751554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.751776 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.752370 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:41.752373 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:41.752737 (+ 364us) client_negotiation.cc:769] Sending connection context
0812 02:37:41.752981 (+ 244us) client_negotiation.cc:240] Negotiation successful
0812 02:37:41.753392 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8504,"mutex_wait_us":142,"spinlock_wait_cycles":36864,"thread_start_us":1359,"threads_started":1}
I20260812 02:37:41.754720 15486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.724607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:33229 (local address 127.3.127.130:41511)
0812 02:37:41.737873 (+ 13266us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.737886 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.738012 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.738643 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.738653 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.739418 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:41.740521 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.740544 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.741888 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.741891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.743624 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.743637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.744596 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.744602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.744740 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.745952 (+ 1212us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:41.745967 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:41.748503 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:41.750940 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.750943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.750945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.751283 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.751601 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.751603 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.751605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.751787 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.752328 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:41.752335 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:41.752737 (+ 402us) client_negotiation.cc:769] Sending connection context
0812 02:37:41.753580 (+ 843us) client_negotiation.cc:240] Negotiation successful
0812 02:37:41.753739 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13167,"mutex_wait_us":3798,"spinlock_wait_cycles":29568,"thread_start_us":2517,"threads_started":1}
I20260812 02:37:41.754720 15487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.738456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.130:52497)
0812 02:37:41.739658 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.739673 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.739818 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.740187 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.740190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.740409 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:41.740522 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.740544 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.741786 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.741789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.743637 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.743644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.744452 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.744456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.744737 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.745579 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:41.745600 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:41.748386 (+ 2786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:41.750704 (+ 2318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.750712 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.750736 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.751204 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.751564 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.751567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.751569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.751787 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.752328 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:41.752335 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:41.752737 (+ 402us) client_negotiation.cc:769] Sending connection context
0812 02:37:41.753720 (+ 983us) client_negotiation.cc:240] Negotiation successful
0812 02:37:41.753838 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1133,"mutex_wait_us":268,"spinlock_wait_cycles":110976,"thread_start_us":342,"threads_started":1}
I20260812 02:37:41.755571 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.728131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:52747 (local address 127.3.127.189:43853)
0812 02:37:41.738927 (+ 10796us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.738932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.738950 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:37:41.738996 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.738999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.739048 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.739139 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.741356 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.742110 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.743800 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.744134 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.744212 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.748600 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.748605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.748607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.748638 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.750637 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.751350 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.751354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.751356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.751452 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.751930 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.751933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.751935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.752202 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.752292 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.755277 (+ 2985us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.755412 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10685}
I20260812 02:37:41.756358 15479 heartbeater.cc:347] Connected to a master server at 127.3.127.188:33229
I20260812 02:37:41.756384 15481 heartbeater.cc:347] Connected to a master server at 127.3.127.190:38495
I20260812 02:37:41.756640 15481 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:41.756672 15482 heartbeater.cc:347] Connected to a master server at 127.3.127.189:43853
I20260812 02:37:41.756726 15482 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:41.756644 15479 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:41.757414 15481 heartbeater.cc:511] Master 127.3.127.190:38495 requested a full tablet report, sending...
I20260812 02:37:41.757674 15479 heartbeater.cc:511] Master 127.3.127.188:33229 requested a full tablet report, sending...
I20260812 02:37:41.757414 15482 heartbeater.cc:511] Master 127.3.127.189:43853 requested a full tablet report, sending...
I20260812 02:37:41.758935 14684 ts_manager.cc:194] Registered new tserver with Master: 3737fe1f65fa40a484cc4f7dc3e030ba (127.3.127.130:34357)
I20260812 02:37:41.758997 13692 ts_manager.cc:194] Re-registered known tserver with Master: 3737fe1f65fa40a484cc4f7dc3e030ba (127.3.127.130:34357)
I20260812 02:37:41.759439 14974 ts_manager.cc:194] Registered new tserver with Master: 3737fe1f65fa40a484cc4f7dc3e030ba (127.3.127.130:34357)
I20260812 02:37:41.759752 13692 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:52497
I20260812 02:37:41.765288 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.757213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52516 (local address 127.3.127.188:33229)
0812 02:37:41.757334 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.757338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.761494 (+ 4156us) server_negotiation.cc:407] Connection header received
0812 02:37:41.761700 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.761703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.761758 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.761839 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:41.762150 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.762643 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.764202 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.764651 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.764749 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.764921 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.765032 (+ 111us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.765127 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 02:37:41.765290 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.756838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:33229 (local address 127.0.0.1:52516)
0812 02:37:41.761372 (+ 4534us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.761391 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.761587 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.761890 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.761893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.761915 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:41.762034 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.762042 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.762750 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.762753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.764077 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.764092 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.764784 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.764788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.764884 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.764942 (+ 58us) client_negotiation.cc:769] Sending connection context
0812 02:37:41.765018 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 02:37:41.765103 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4442,"thread_start_us":83,"threads_started":1}
I20260812 02:37:41.766176 14991 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260812 02:37:41.766687 15051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05763676674c4a82b5e510ce078c3b07. Latest consensus state: current_term: 1 leader_uuid: "05763676674c4a82b5e510ce078c3b07" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: NON_VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: true } } }
I20260812 02:37:41.766834 15051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [sys.catalog]: This master's current role is: LEARNER
W20260812 02:37:41.876351 15488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:41.876705 15488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:41.876751 15488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:41.881036 15488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:41.881114 15488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:41.881212 15488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:37:41.886834 15488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.131.keytab
--principal=oryx/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:40167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=43769
--tserver_master_addrs=127.3.127.190:38495,127.3.127.189:43853,127.3.127.188:33229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:41.887979 15488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:41.889027 15488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:41.899891 15500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:41.899992 15497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:41.899886 15496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:41.900353 15488 server_base.cc:1034] running on GCE node
I20260812 02:37:41.900818 15488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:41.901605 15488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:41.904920 15488 hybrid_clock.cc:648] HybridClock initialized: now 1786502261904873 us; error 45 us; skew 500 ppm
Aug 12 02:37:41 dist-test-slave-rmhg krb5kdc[13623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502261, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:41.908370 15488 init.cc:377] Logged in from keytab as oryx/127.3.127.131@KRBTEST.COM (short username oryx)
I20260812 02:37:41.908488 14767 catalog_manager.cc:1259] Loaded cluster ID: 1edca9d490854f83b1c360530e4749c0
I20260812 02:37:41.908545 14767 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: loading cluster ID for follower catalog manager: success
I20260812 02:37:41.909688 15488 webserver.cc:460] Webserver started at http://127.3.127.131:43769/ using document root <none> and password file <none>
I20260812 02:37:41.910387 15488 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:41.910507 15488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:41.910861 14767 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: acquiring CA information for follower catalog manager: success
I20260812 02:37:41.912353 14767 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:37:41.915252 15488 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 02:37:41.917631 15514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:41.918942 15488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:41.919057 15488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "6f0f40aab58f4476a0c404dfce26053c"
format_stamp: "Formatted at 2026-08-12 02:37:40 on dist-test-slave-rmhg"
server_key: "1d86c97aa9cefa0b7e4ae0092e6ef52b"
server_key_iv: "bbcb8fa2f6261d0cc2e76bfcab9dfdfb"
server_key_version: "encryptionkey@0"
I20260812 02:37:41.919662 15488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:41.934304 15488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:41.937520 15488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:41.937704 15488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:41.938297 15488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:41.939184 15488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:41.939232 15488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:41.939273 15488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:41.939292 15488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:41.948273 15488 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:40167
I20260812 02:37:41.948298 15688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:40167 every 8 connection(s)
I20260812 02:37:41.949465 15488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:37:41.950743 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 15488
Aug 12 02:37:41 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502261, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.188@KRBTEST.COM
Aug 12 02:37:41 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502261, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.189@KRBTEST.COM
Aug 12 02:37:41 dist-test-slave-rmhg krb5kdc[13623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502261, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM
I20260812 02:37:41.968763 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.953129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47058 (local address 127.3.127.190:38495)
0812 02:37:41.953392 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.953396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.953663 (+ 267us) server_negotiation.cc:407] Connection header received
0812 02:37:41.953885 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.953889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.953949 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.954014 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.955259 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.956077 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.957313 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.957668 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.957755 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.961237 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.961241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.961244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.961279 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.963606 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.967012 (+ 3406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.967016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.967017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.967347 (+ 330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.967682 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.967686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.967687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.967932 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.968049 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.968423 (+ 374us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.968566 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"spinlock_wait_cycles":1920,"thread_start_us":78,"threads_started":1}
I20260812 02:37:41.970525 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.953367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60228 (local address 127.3.127.189:43853)
0812 02:37:41.954585 (+ 1218us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.954589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.955154 (+ 565us) server_negotiation.cc:407] Connection header received
0812 02:37:41.956429 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.956433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.956480 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.956559 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.957505 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.958292 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.959387 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.959734 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.959833 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.962292 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.962296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.962299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.962330 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.966381 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.967177 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.967181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.967183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.967284 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.967607 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.967611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.967613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.967881 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.968007 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.968652 (+ 645us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.970351 (+ 1699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1146,"thread_start_us":89,"threads_started":1}
I20260812 02:37:41.970837 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.953707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:60913 (local address 127.3.127.188:33229)
0812 02:37:41.954021 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.954026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.954426 (+ 400us) server_negotiation.cc:407] Connection header received
0812 02:37:41.956040 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.956044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.956098 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.956147 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.958563 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.959299 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.961070 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.961406 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.961485 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.965406 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.965410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.965413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.965444 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.967991 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.968892 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.968895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.968897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.968985 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.969465 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.969469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.969470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.969716 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.969790 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.970555 (+ 765us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.970690 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242}
I20260812 02:37:41.971745 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.952771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:47313 (local address 127.3.127.190:38495)
0812 02:37:41.953004 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.953008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.954071 (+ 1063us) server_negotiation.cc:407] Connection header received
0812 02:37:41.955444 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.955447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.955498 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.955548 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.958175 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.958916 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.960870 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.961217 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.961287 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.967015 (+ 5728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.967018 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.967021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.967058 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.969494 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.970078 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.970081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.970083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.970160 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.970456 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.970459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.970461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.970668 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.970750 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.971321 (+ 571us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.971439 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"spinlock_wait_cycles":2432}
I20260812 02:37:41.972465 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.953804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52522 (local address 127.3.127.188:33229)
0812 02:37:41.954129 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.954133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.954855 (+ 722us) server_negotiation.cc:407] Connection header received
0812 02:37:41.954913 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.954916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.954972 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.955065 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.960373 (+ 5308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.961125 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.963664 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.964100 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.964193 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.966344 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.966348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.966351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.966380 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.970230 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.971028 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.971031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.971033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.971116 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.971439 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.971443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.971444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.971723 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.971824 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.972170 (+ 346us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.972297 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":91,"threads_started":1}
I20260812 02:37:41.973682 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.952074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.131:47313)
0812 02:37:41.953902 (+ 1828us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.953969 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.955194 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.956006 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.956016 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.957900 (+ 1884us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:41.958031 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.958038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.959092 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.959095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.960721 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.960733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.961487 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.961491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.961737 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.964174 (+ 2437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:41.964189 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:41.966410 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:41.969758 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.969761 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.969763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.969971 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.970263 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.970266 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.970268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.970356 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.970813 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:41.970817 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:41.970910 (+ 93us) client_negotiation.cc:769] Sending connection context
0812 02:37:41.971933 (+ 1023us) client_negotiation.cc:240] Negotiation successful
0812 02:37:41.972348 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1744,"mutex_wait_us":183,"spinlock_wait_cycles":75392,"thread_start_us":170,"threads_started":1}
I20260812 02:37:41.974013 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.953304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:33229 (local address 127.3.127.131:60913)
0812 02:37:41.954323 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.954338 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.955850 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.956287 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.956290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.958295 (+ 2005us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:41.958419 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.958426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.959433 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.959436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.960931 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.960944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.961677 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.961681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.961797 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.962415 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:41.962436 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:41.965181 (+ 2745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:41.968193 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.968200 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.968221 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.968774 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.969103 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.969106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.969108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.969346 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.969878 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:41.969885 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:41.970265 (+ 380us) client_negotiation.cc:769] Sending connection context
0812 02:37:41.971890 (+ 1625us) client_negotiation.cc:240] Negotiation successful
0812 02:37:41.972431 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"mutex_wait_us":264,"thread_start_us":82,"threads_started":1}
I20260812 02:37:41.974225 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.952628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:55537 (local address 127.3.127.189:43853)
0812 02:37:41.952914 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.952919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.954070 (+ 1151us) server_negotiation.cc:407] Connection header received
0812 02:37:41.955444 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.955447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.955498 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.955549 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:41.957855 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.958936 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.962812 (+ 3876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.963155 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.963240 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.967205 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:41.967209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:41.967212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:41.967242 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:41.969056 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.969556 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.969560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.969561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.969660 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:41.973251 (+ 3591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:41.973255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:41.973256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:41.973521 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:41.973606 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:41.973954 (+ 348us) server_negotiation.cc:299] Negotiation successful
0812 02:37:41.974081 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260812 02:37:41.974664 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.952074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:43853 (local address 127.3.127.131:55537)
0812 02:37:41.953903 (+ 1829us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.953969 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.955192 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.956006 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.956016 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.956779 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:41.957607 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.957629 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.959092 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.959096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.962649 (+ 3553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.962661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.963425 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.963429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.963531 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:41.964004 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:41.964020 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:41.966848 (+ 2828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:41.969199 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.969202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.969204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.969437 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.972924 (+ 3487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:41.972927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:41.972929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:41.973087 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:41.973665 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:41.973667 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:41.973761 (+ 94us) client_negotiation.cc:769] Sending connection context
0812 02:37:41.974360 (+ 599us) client_negotiation.cc:240] Negotiation successful
0812 02:37:41.974507 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1594,"mutex_wait_us":2197,"spinlock_wait_cycles":18816,"thread_start_us":145,"threads_started":1}
I20260812 02:37:41.975236 15689 heartbeater.cc:347] Connected to a master server at 127.3.127.188:33229
I20260812 02:37:41.975296 15691 heartbeater.cc:347] Connected to a master server at 127.3.127.189:43853
I20260812 02:37:41.975486 15689 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:41.975505 15691 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:41.975983 15690 heartbeater.cc:347] Connected to a master server at 127.3.127.190:38495
I20260812 02:37:41.976073 15690 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:41.976099 15691 heartbeater.cc:511] Master 127.3.127.189:43853 requested a full tablet report, sending...
I20260812 02:37:41.976284 15689 heartbeater.cc:511] Master 127.3.127.188:33229 requested a full tablet report, sending...
I20260812 02:37:41.976310 15690 heartbeater.cc:511] Master 127.3.127.190:38495 requested a full tablet report, sending...
I20260812 02:37:41.973119 13692 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 02:37:41.977303 14974 ts_manager.cc:194] Registered new tserver with Master: 6f0f40aab58f4476a0c404dfce26053c (127.3.127.131:40167)
I20260812 02:37:41.977308 13691 ts_manager.cc:194] Re-registered known tserver with Master: 6f0f40aab58f4476a0c404dfce26053c (127.3.127.131:40167)
I20260812 02:37:41.977347 14684 ts_manager.cc:194] Registered new tserver with Master: 6f0f40aab58f4476a0c404dfce26053c (127.3.127.131:40167)
I20260812 02:37:41.978057 13691 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:47313
W20260812 02:37:41.978288 13692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:37:41.981308 14933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05763676674c4a82b5e510ce078c3b07" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: NON_VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: true } } }
I20260812 02:37:41.981462 14933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:41.984819 15718 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07: attempting to promote NON_VOTER 0b2ff8a5d080404dbddd48b9a04d9766 to VOTER
I20260812 02:37:41.985229 15718 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false } }
I20260812 02:37:41.986279 14991 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 1 LEARNER]: Refusing update from remote peer 05763676674c4a82b5e510ce078c3b07: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260812 02:37:41.986285 14699 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 1 FOLLOWER]: Refusing update from remote peer 05763676674c4a82b5e510ce078c3b07: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260812 02:37:41.986720 15717 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260812 02:37:41.987730 14933 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260812 02:37:41.989338 15718 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0b2ff8a5d080404dbddd48b9a04d9766 (127.3.127.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false } } }
I20260812 02:37:41.989849 14991 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0b2ff8a5d080404dbddd48b9a04d9766 (127.3.127.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false } } }
I20260812 02:37:41.990828 15718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "05763676674c4a82b5e510ce078c3b07" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false } } }
I20260812 02:37:41.990927 15718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:41.990989 15717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05763676674c4a82b5e510ce078c3b07" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false } } }
I20260812 02:37:41.991036 14933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05763676674c4a82b5e510ce078c3b07" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false } } }
I20260812 02:37:41.991096 15717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:41.991122 14933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:41.991930 15051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05763676674c4a82b5e510ce078c3b07" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false } } }
I20260812 02:37:41.992049 15051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:37:41.992726 14699 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0b2ff8a5d080404dbddd48b9a04d9766 (127.3.127.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false } } }
I20260812 02:37:41.997061 15731 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05763676674c4a82b5e510ce078c3b07" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false } } }
I20260812 02:37:41.997327 15731 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:37:42.000793 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.995721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:37785 (local address 127.0.0.1:60910)
0812 02:37:41.996285 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.996299 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.996394 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.996952 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.996955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.996976 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:41.997096 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.997103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.998615 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.998618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:41.999516 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.999524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.000334 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.000338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:42.000429 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:42.000453 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 02:37:42.000514 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 02:37:42.000605 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":1}
I20260812 02:37:42.000824 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.995906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60910 (local address 127.3.127.129:37785)
0812 02:37:41.996242 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.996251 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.996371 (+ 120us) server_negotiation.cc:407] Connection header received
0812 02:37:41.996518 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.996525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.996789 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.997306 (+ 517us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:41.997464 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:41.998508 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:41.999629 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.000210 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.000313 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:42.000486 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:42.000543 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 02:37:42.000643 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":115,"threads_started":1}
I20260812 02:37:42.004212 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.995922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:34357 (local address 127.0.0.1:46314)
0812 02:37:41.996784 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.996799 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.996930 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.999253 (+ 2323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.999256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.999275 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:41.999380 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.999386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.001228 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.001232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:42.002442 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:42.002454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.003755 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.003759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:42.003854 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:42.003884 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 02:37:42.003972 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 02:37:42.004053 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260812 02:37:42.004408 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.996587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.3.127.130:34357)
0812 02:37:41.998699 (+ 2112us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.998740 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.998766 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:37:41.998843 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.998847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.999047 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.999542 (+ 495us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:41.999703 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.001056 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.002608 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.003618 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.003735 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:42.003891 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:42.004027 (+ 136us) server_negotiation.cc:299] Negotiation successful
0812 02:37:42.004173 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1996,"thread_start_us":117,"threads_started":1}
I20260812 02:37:42.004200 15165 tablet_service.cc:1467] Processing CreateTablet for tablet d531b44ffb0c4e0dafd385cdef6a68d7 (DEFAULT_TABLE table=test-table [id=51c41ac9c9964d95ad7b1ad2615ec1df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:42.005573 15165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d531b44ffb0c4e0dafd385cdef6a68d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:42.005605 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.996258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:40167 (local address 127.0.0.1:46150)
0812 02:37:41.997210 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:41.997223 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:41.998391 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:41.999029 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:41.999032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:41.999052 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:41.999145 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:41.999151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.001495 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.001499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:42.002972 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:42.002983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.004823 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.004827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:42.004919 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:42.004944 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 02:37:42.005003 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 02:37:42.005072 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":82,"threads_started":1}
I20260812 02:37:42.006305 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:41.996648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46150 (local address 127.3.127.131:40167)
0812 02:37:41.997692 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:41.997718 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:41.997746 (+ 28us) server_negotiation.cc:407] Connection header received
0812 02:37:41.998569 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:41.998577 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:41.998836 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:41.999042 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:41.999252 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.000682 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.003133 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.004050 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.004160 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:42.005937 (+ 1777us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:42.005997 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 02:37:42.006125 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":926,"thread_start_us":121,"threads_started":1}
I20260812 02:37:42.007912 15382 tablet_service.cc:1467] Processing CreateTablet for tablet d531b44ffb0c4e0dafd385cdef6a68d7 (DEFAULT_TABLE table=test-table [id=51c41ac9c9964d95ad7b1ad2615ec1df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:42.009315 15382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d531b44ffb0c4e0dafd385cdef6a68d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:42.009335 15584 tablet_service.cc:1467] Processing CreateTablet for tablet d531b44ffb0c4e0dafd385cdef6a68d7 (DEFAULT_TABLE table=test-table [id=51c41ac9c9964d95ad7b1ad2615ec1df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:42.010394 15584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d531b44ffb0c4e0dafd385cdef6a68d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:42.017912 15741 tablet_bootstrap.cc:492] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434: Bootstrap starting.
I20260812 02:37:42.019897 15742 tablet_bootstrap.cc:492] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c: Bootstrap starting.
I20260812 02:37:42.021247 15741 tablet_bootstrap.cc:654] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:42.022648 15742 tablet_bootstrap.cc:654] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:42.022908 15741 log.cc:824] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:42.024080 15742 log.cc:824] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:42.025663 15743 tablet_bootstrap.cc:492] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba: Bootstrap starting.
I20260812 02:37:42.026192 15741 tablet_bootstrap.cc:492] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434: No bootstrap required, opened a new log
I20260812 02:37:42.026423 15741 ts_tablet_manager.cc:1397] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260812 02:37:42.029451 15743 tablet_bootstrap.cc:654] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:42.031118 15742 tablet_bootstrap.cc:492] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c: No bootstrap required, opened a new log
I20260812 02:37:42.031098 15743 log.cc:824] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:42.031339 15742 ts_tablet_manager.cc:1397] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20260812 02:37:42.033527 15741 raft_consensus.cc:348] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } } peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } }
I20260812 02:37:42.033852 15741 raft_consensus.cc:374] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:42.033916 15741 raft_consensus.cc:729] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 430e867652ed41ff8cda09459a710434, State: Initialized, Role: FOLLOWER
I20260812 02:37:42.034096 15743 tablet_bootstrap.cc:492] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba: No bootstrap required, opened a new log
I20260812 02:37:42.034303 15743 ts_tablet_manager.cc:1397] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260812 02:37:42.034480 15741 consensus_queue.cc:260] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } } peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } }
I20260812 02:37:42.035249 15264 heartbeater.cc:503] Master 127.3.127.190:38495 was elected leader, sending a full tablet report...
I20260812 02:37:42.035725 15741 ts_tablet_manager.cc:1428] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20260812 02:37:42.037480 15742 raft_consensus.cc:348] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } } peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } }
I20260812 02:37:42.037765 15742 raft_consensus.cc:374] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:42.037820 15742 raft_consensus.cc:729] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f0f40aab58f4476a0c404dfce26053c, State: Initialized, Role: FOLLOWER
I20260812 02:37:42.038307 15742 consensus_queue.cc:260] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } } peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } }
I20260812 02:37:42.039417 15742 ts_tablet_manager.cc:1428] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260812 02:37:42.040854 15690 heartbeater.cc:503] Master 127.3.127.190:38495 was elected leader, sending a full tablet report...
I20260812 02:37:42.043942 15743 raft_consensus.cc:348] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } } peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } }
I20260812 02:37:42.044265 15743 raft_consensus.cc:374] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:42.044335 15743 raft_consensus.cc:729] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3737fe1f65fa40a484cc4f7dc3e030ba, State: Initialized, Role: FOLLOWER
I20260812 02:37:42.044900 15743 consensus_queue.cc:260] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } } peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } }
I20260812 02:37:42.045855 15481 heartbeater.cc:503] Master 127.3.127.190:38495 was elected leader, sending a full tablet report...
I20260812 02:37:42.046151 15743 ts_tablet_manager.cc:1428] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba: Time spent starting tablet: real 0.012s user 0.002s sys 0.007s
I20260812 02:37:42.073612 15752 raft_consensus.cc:482] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:37:42.073902 15752 raft_consensus.cc:504] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } } peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } }
I20260812 02:37:42.074393 15753 raft_consensus.cc:482] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:37:42.074640 15753 raft_consensus.cc:504] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } } peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } }
I20260812 02:37:42.075563 15752 leader_election.cc:288] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3737fe1f65fa40a484cc4f7dc3e030ba (127.3.127.130:34357), 6f0f40aab58f4476a0c404dfce26053c (127.3.127.131:40167)
I20260812 02:37:42.075934 15753 leader_election.cc:288] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 430e867652ed41ff8cda09459a710434 (127.3.127.129:37785), 3737fe1f65fa40a484cc4f7dc3e030ba (127.3.127.130:34357)
I20260812 02:37:42.081434 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:42.076173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:37785 (local address 127.3.127.131:52971)
0812 02:37:42.076409 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:42.076424 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:42.076520 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:42.077200 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:42.077203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:42.077222 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:42.077475 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:42.077485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.078518 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.078522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:42.079768 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:42.079780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.080712 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.080715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:42.080901 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:42.080928 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 02:37:42.081003 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 02:37:42.081086 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":1}
I20260812 02:37:42.081794 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:42.076098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:34247 (local address 127.3.127.130:34357)
0812 02:37:42.076249 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:42.076253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:42.077079 (+ 826us) server_negotiation.cc:407] Connection header received
0812 02:37:42.077292 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:42.077295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:42.077338 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:42.077387 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:42.077979 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.078654 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.080458 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.081019 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.081101 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:42.081176 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:42.081577 (+ 401us) server_negotiation.cc:299] Negotiation successful
0812 02:37:42.081673 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 02:37:42.082201 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:42.076283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:34357 (local address 127.3.127.131:52617)
0812 02:37:42.076454 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:42.076468 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:42.076553 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:42.077290 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:42.077293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:42.077310 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:42.077505 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:42.077513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.079262 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.079265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:42.080677 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:42.080689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.081744 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.081748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:42.081916 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:42.081949 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 02:37:42.081992 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 02:37:42.082069 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":2}
I20260812 02:37:42.082239 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:42.076723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:52971 (local address 127.3.127.129:37785)
0812 02:37:42.076830 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:42.076835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:42.076851 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:37:42.076899 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:42.076902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:42.076951 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:42.077026 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:42.077595 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.078399 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.079920 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.080497 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.080597 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:42.080652 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:42.082044 (+ 1392us) server_negotiation.cc:299] Negotiation successful
0812 02:37:42.082105 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":35}
I20260812 02:37:42.082314 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:42.075847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:34357 (local address 127.3.127.129:34247)
0812 02:37:42.076961 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:42.076974 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:42.077173 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:42.077526 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:42.077528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:42.077547 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:42.077845 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:42.077856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.078791 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.078795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:42.080323 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:42.080333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.081270 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.081273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:42.081423 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:42.081451 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 02:37:42.081982 (+ 531us) client_negotiation.cc:240] Negotiation successful
0812 02:37:42.082067 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1056,"mutex_wait_us":90,"thread_start_us":225,"threads_started":1}
I20260812 02:37:42.082625 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:42.077840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:49939 (local address 127.3.127.131:40167)
0812 02:37:42.077951 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:42.077955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:42.077968 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:37:42.078006 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:42.078009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:42.078047 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:42.078123 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:42.078651 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.079402 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.081230 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.081864 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.081946 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:42.081989 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:42.082448 (+ 459us) server_negotiation.cc:299] Negotiation successful
0812 02:37:42.082524 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":48}
I20260812 02:37:42.082698 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:42.076392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:40167 (local address 127.3.127.129:49939)
0812 02:37:42.077065 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:42.077078 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:42.077234 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:42.078218 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:42.078222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:42.078239 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:42.078514 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:42.078524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.079540 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.079543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:42.081061 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:42.081072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.082092 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.082096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:42.082318 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:42.082347 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 02:37:42.082421 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 02:37:42.082493 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":1,"thread_start_us":185,"threads_started":1}
I20260812 02:37:42.083057 15195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d531b44ffb0c4e0dafd385cdef6a68d7" candidate_uuid: "6f0f40aab58f4476a0c404dfce26053c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "430e867652ed41ff8cda09459a710434" is_pre_election: true
I20260812 02:37:42.083281 15408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d531b44ffb0c4e0dafd385cdef6a68d7" candidate_uuid: "430e867652ed41ff8cda09459a710434" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" is_pre_election: true
I20260812 02:37:42.083281 15619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d531b44ffb0c4e0dafd385cdef6a68d7" candidate_uuid: "430e867652ed41ff8cda09459a710434" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f0f40aab58f4476a0c404dfce26053c" is_pre_election: true
I20260812 02:37:42.083340 15195 raft_consensus.cc:2454] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f0f40aab58f4476a0c404dfce26053c in term 0.
I20260812 02:37:42.083509 15619 raft_consensus.cc:2454] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 430e867652ed41ff8cda09459a710434 in term 0.
I20260812 02:37:42.083576 15408 raft_consensus.cc:2454] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 430e867652ed41ff8cda09459a710434 in term 0.
I20260812 02:37:42.083997 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:42.076487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:52617 (local address 127.3.127.130:34357)
0812 02:37:42.077004 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:42.077013 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:42.077029 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:37:42.077085 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:42.077088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:42.077140 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:42.077634 (+ 494us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:42.077829 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.079137 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.080833 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.081612 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.083631 (+ 2019us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:42.083696 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:42.083732 (+ 36us) server_negotiation.cc:299] Negotiation successful
0812 02:37:42.083799 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":199,"threads_started":1}
I20260812 02:37:42.084074 15515 leader_election.cc:302] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 430e867652ed41ff8cda09459a710434, 6f0f40aab58f4476a0c404dfce26053c; no voters:
I20260812 02:37:42.084097 15089 leader_election.cc:302] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 430e867652ed41ff8cda09459a710434, 6f0f40aab58f4476a0c404dfce26053c; no voters:
I20260812 02:37:42.084492 15408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d531b44ffb0c4e0dafd385cdef6a68d7" candidate_uuid: "6f0f40aab58f4476a0c404dfce26053c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" is_pre_election: true
I20260812 02:37:42.084517 15752 raft_consensus.cc:2781] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:37:42.084610 15752 raft_consensus.cc:482] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:37:42.084612 15408 raft_consensus.cc:2454] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f0f40aab58f4476a0c404dfce26053c in term 0.
I20260812 02:37:42.084666 15752 raft_consensus.cc:3037] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:42.084724 15753 raft_consensus.cc:2781] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:37:42.084808 15753 raft_consensus.cc:482] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:37:42.084849 15753 raft_consensus.cc:3037] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:42.086701 15752 raft_consensus.cc:504] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } } peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } }
I20260812 02:37:42.086701 15753 raft_consensus.cc:504] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } } peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } }
I20260812 02:37:42.087085 15752 leader_election.cc:288] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [CANDIDATE]: Term 1 election: Requested vote from peers 3737fe1f65fa40a484cc4f7dc3e030ba (127.3.127.130:34357), 6f0f40aab58f4476a0c404dfce26053c (127.3.127.131:40167)
I20260812 02:37:42.087128 15753 leader_election.cc:288] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c [CANDIDATE]: Term 1 election: Requested vote from peers 430e867652ed41ff8cda09459a710434 (127.3.127.129:37785), 3737fe1f65fa40a484cc4f7dc3e030ba (127.3.127.130:34357)
I20260812 02:37:42.087525 15195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d531b44ffb0c4e0dafd385cdef6a68d7" candidate_uuid: "6f0f40aab58f4476a0c404dfce26053c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "430e867652ed41ff8cda09459a710434"
I20260812 02:37:42.087541 15408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d531b44ffb0c4e0dafd385cdef6a68d7" candidate_uuid: "430e867652ed41ff8cda09459a710434" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba"
I20260812 02:37:42.087525 15619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d531b44ffb0c4e0dafd385cdef6a68d7" candidate_uuid: "430e867652ed41ff8cda09459a710434" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f0f40aab58f4476a0c404dfce26053c"
I20260812 02:37:42.087657 15406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d531b44ffb0c4e0dafd385cdef6a68d7" candidate_uuid: "6f0f40aab58f4476a0c404dfce26053c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba"
I20260812 02:37:42.087682 15408 raft_consensus.cc:3037] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:42.087756 15195 raft_consensus.cc:2379] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6f0f40aab58f4476a0c404dfce26053c in current term 1: Already voted for candidate 430e867652ed41ff8cda09459a710434 in this term.
I20260812 02:37:42.087756 15619 raft_consensus.cc:2379] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 430e867652ed41ff8cda09459a710434 in current term 1: Already voted for candidate 6f0f40aab58f4476a0c404dfce26053c in this term.
I20260812 02:37:42.088315 15518 leader_election.cc:302] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f0f40aab58f4476a0c404dfce26053c; no voters: 3737fe1f65fa40a484cc4f7dc3e030ba, 430e867652ed41ff8cda09459a710434
I20260812 02:37:42.088490 15753 raft_consensus.cc:2726] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260812 02:37:42.089398 15408 raft_consensus.cc:2454] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 430e867652ed41ff8cda09459a710434 in term 1.
I20260812 02:37:42.089746 15089 leader_election.cc:302] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3737fe1f65fa40a484cc4f7dc3e030ba, 430e867652ed41ff8cda09459a710434; no voters: 6f0f40aab58f4476a0c404dfce26053c
I20260812 02:37:42.089920 15752 raft_consensus.cc:2781] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:42.090170 15752 raft_consensus.cc:686] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [term 1 LEADER]: Becoming Leader. State: Replica: 430e867652ed41ff8cda09459a710434, State: Running, Role: LEADER
I20260812 02:37:42.090513 15752 consensus_queue.cc:237] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } } peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } }
I20260812 02:37:42.094166 13692 catalog_manager.cc:5495] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 reported cstate change: term changed from 0 to 1, leader changed from <none> to 430e867652ed41ff8cda09459a710434 (127.3.127.129). New cstate: current_term: 1 leader_uuid: "430e867652ed41ff8cda09459a710434" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } health_report { overall_health: UNKNOWN } } }
I20260812 02:37:42.126219 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:42.122216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60918 (local address 127.3.127.129:37785)
0812 02:37:42.122389 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:42.122392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:42.122404 (+ 12us) server_negotiation.cc:407] Connection header received
0812 02:37:42.122486 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:42.122488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:42.122548 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:42.122640 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:42.123061 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.123609 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.124534 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.124757 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.124850 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:42.125330 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:42.125795 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:42.125912 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:42.125994 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 02:37:42.126060 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 02:37:42.133423 15619 raft_consensus.cc:1261] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c [term 1 FOLLOWER]: Refusing update from remote peer 430e867652ed41ff8cda09459a710434: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:42.133423 15408 raft_consensus.cc:1261] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba [term 1 FOLLOWER]: Refusing update from remote peer 430e867652ed41ff8cda09459a710434: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:42.134248 15752 consensus_queue.cc:1035] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:42.134557 15760 consensus_queue.cc:1035] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:42.151229 15768 mvcc.cc:205] Tried to move back new op lower bound from 7317513265689600000 to 7317513265525555200. Current Snapshot: MvccSnapshot[applied={T|T < 7317513265689600000 or (T in {7317513265689600000})}]
I20260812 02:37:42.160305 13691 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:42.160493 13691 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:42.165695 13691 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05763676674c4a82b5e510ce078c3b07: Sending DeleteTablet for 3 replicas of tablet d531b44ffb0c4e0dafd385cdef6a68d7
I20260812 02:37:42.167119 15584 tablet_service.cc:1514] Processing DeleteTablet for tablet d531b44ffb0c4e0dafd385cdef6a68d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:42 UTC) from {username='oryx', principal='oryx/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:46150
I20260812 02:37:42.167141 15382 tablet_service.cc:1514] Processing DeleteTablet for tablet d531b44ffb0c4e0dafd385cdef6a68d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:42 UTC) from {username='oryx', principal='oryx/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:46314
I20260812 02:37:42.168087 15781 tablet_replica.cc:330] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba: stopping tablet replica
I20260812 02:37:42.168107 15165 tablet_service.cc:1514] Processing DeleteTablet for tablet d531b44ffb0c4e0dafd385cdef6a68d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:42 UTC) from {username='oryx', principal='oryx/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:60910
I20260812 02:37:42.168509 15781 raft_consensus.cc:2229] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:42.168889 15781 raft_consensus.cc:2258] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:42.168880 15783 tablet_replica.cc:330] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c: stopping tablet replica
I20260812 02:37:42.169317 15787 tablet_replica.cc:330] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434: stopping tablet replica
I20260812 02:37:42.169631 15787 raft_consensus.cc:2229] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:37:42.170056 15787 raft_consensus.cc:2258] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:42.170138 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 13647
I20260812 02:37:42.176713 15783 raft_consensus.cc:2229] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:42.177389 15783 raft_consensus.cc:2258] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:42.181087 15781 ts_tablet_manager.cc:1905] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:42.181087 15783 ts_tablet_manager.cc:1905] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:42.182478 15787 ts_tablet_manager.cc:1905] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:42.184875 15783 ts_tablet_manager.cc:1918] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:37:42.185020 15783 log.cc:1196] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/d531b44ffb0c4e0dafd385cdef6a68d7
I20260812 02:37:42.185397 15783 ts_tablet_manager.cc:1939] T d531b44ffb0c4e0dafd385cdef6a68d7 P 6f0f40aab58f4476a0c404dfce26053c: Deleting consensus metadata
I20260812 02:37:42.188171 15781 ts_tablet_manager.cc:1918] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:37:42.188313 15781 log.cc:1196] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/d531b44ffb0c4e0dafd385cdef6a68d7
I20260812 02:37:42.188524 15787 ts_tablet_manager.cc:1918] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:37:42.188683 15781 ts_tablet_manager.cc:1939] T d531b44ffb0c4e0dafd385cdef6a68d7 P 3737fe1f65fa40a484cc4f7dc3e030ba: Deleting consensus metadata
I20260812 02:37:42.188683 15787 log.cc:1196] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-0/wal/wals/d531b44ffb0c4e0dafd385cdef6a68d7
I20260812 02:37:42.189062 15787 ts_tablet_manager.cc:1939] T d531b44ffb0c4e0dafd385cdef6a68d7 P 430e867652ed41ff8cda09459a710434: Deleting consensus metadata
W20260812 02:37:42.189829 15521 connection.cc:535] server connection from 127.0.0.1:46150 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 02:37:42.190337 15305 connection.cc:535] server connection from 127.0.0.1:46314 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 02:37:42.191004 15089 connection.cc:535] server connection from 127.0.0.1:60910 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 02:37:42.195014 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:42.167749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60240 (local address 127.3.127.189:43853)
0812 02:37:42.167865 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:42.167870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:42.169371 (+ 1501us) server_negotiation.cc:407] Connection header received
0812 02:37:42.180095 (+ 10724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:42.180099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:42.180164 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:42.180219 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:42.185330 (+ 5111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.186131 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.187644 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.187994 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.188082 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:42.191028 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:42.191033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:42.191036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:42.191072 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:42.192873 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:42.193479 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:42.193483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:42.193485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:42.193592 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:42.193909 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:42.193913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:42.193914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:42.194194 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:42.194326 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:42.194697 (+ 371us) server_negotiation.cc:299] Negotiation successful
0812 02:37:42.194834 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 02:37:42.205312 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:42.194856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52542 (local address 127.3.127.188:33229)
0812 02:37:42.194973 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:42.194978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:42.195574 (+ 596us) server_negotiation.cc:407] Connection header received
0812 02:37:42.195750 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:42.195754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:42.195804 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:42.195854 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:42.196834 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.197688 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.198932 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.199289 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.199362 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:42.201413 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:42.201417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:42.201420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:42.201448 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:42.203283 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:42.203772 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:42.203776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:42.203777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:42.203865 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:42.204152 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:42.204156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:42.204169 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:42.204386 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:42.204503 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:42.204907 (+ 404us) server_negotiation.cc:299] Negotiation successful
0812 02:37:42.205039 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 02:37:42.205312 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:42.194277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60252 (local address 127.3.127.189:43853)
0812 02:37:42.194427 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:42.194432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:42.195068 (+ 636us) server_negotiation.cc:407] Connection header received
0812 02:37:42.195266 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:42.195269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:42.195316 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:42.195365 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:42.196298 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.197038 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.198385 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.198794 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.198876 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:42.200986 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:42.200991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:42.200994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:42.201026 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:42.202785 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:42.203635 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:42.203639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:42.203641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:42.203745 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:42.204072 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:42.204075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:42.204077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:42.204339 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:42.204452 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:42.204870 (+ 418us) server_negotiation.cc:299] Negotiation successful
0812 02:37:42.205034 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 02:37:42.206570 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:42.168273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52534 (local address 127.3.127.188:33229)
0812 02:37:42.168387 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:42.168392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:42.196752 (+ 28360us) server_negotiation.cc:407] Connection header received
0812 02:37:42.196911 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:42.196914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:42.196963 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:42.197042 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:42.198238 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.198979 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.200030 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:42.200415 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:42.200494 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:42.202055 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:42.202059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:42.202062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:42.202092 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:42.203957 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:42.204586 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:42.204589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:42.204591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:42.204695 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:42.205035 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:42.205039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:42.205041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:42.205760 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:42.205887 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:42.206283 (+ 396us) server_negotiation.cc:299] Negotiation successful
0812 02:37:42.206418 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 02:37:43.150007 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:43.149248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.130:51417)
0812 02:37:43.149765 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:43.149874 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":92,"threads_started":1}
W20260812 02:37:43.150748 15481 heartbeater.cc:644] Failed to heartbeat to 127.3.127.190:38495 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
I20260812 02:37:43.151533 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:43.150854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.131:52371)
0812 02:37:43.151317 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:43.151414 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":92,"threads_started":1}
W20260812 02:37:43.151852 15690 heartbeater.cc:644] Failed to heartbeat to 127.3.127.190:38495 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
I20260812 02:37:43.151978 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:43.151471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.129:60923)
0812 02:37:43.151816 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:43.151893 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":104,"threads_started":1}
W20260812 02:37:43.152254 15264 heartbeater.cc:644] Failed to heartbeat to 127.3.127.190:38495 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
I20260812 02:37:43.666491 15804 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 05763676674c4a82b5e510ce078c3b07)
I20260812 02:37:43.666495 15805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 05763676674c4a82b5e510ce078c3b07)
I20260812 02:37:43.666754 15804 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false } }
I20260812 02:37:43.666754 15805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false } }
I20260812 02:37:43.668001 15805 leader_election.cc:288] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05763676674c4a82b5e510ce078c3b07 (127.3.127.190:38495), dd62f6975b334750bba9d180a80ab0cd (127.3.127.189:43853)
I20260812 02:37:43.667989 15804 leader_election.cc:288] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05763676674c4a82b5e510ce078c3b07 (127.3.127.190:38495), 0b2ff8a5d080404dbddd48b9a04d9766 (127.3.127.188:33229)
I20260812 02:37:43.668810 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:43.668361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47564)
0812 02:37:43.668621 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:43.668704 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":68,"threads_started":1}
I20260812 02:37:43.668983 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:43.668209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47562)
0812 02:37:43.668682 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:43.668763 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":80,"threads_started":1}
W20260812 02:37:43.668987 14954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
W20260812 02:37:43.669327 14650 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
I20260812 02:37:43.670058 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:43.669801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47576)
0812 02:37:43.669934 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:43.669984 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260812 02:37:43.670218 14954 leader_election.cc:334] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 05763676674c4a82b5e510ce078c3b07 (127.3.127.190:38495): Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
I20260812 02:37:43.671203 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:43.670957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47582)
0812 02:37:43.671092 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:43.671143 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260812 02:37:43.671304 14650 leader_election.cc:334] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 05763676674c4a82b5e510ce078c3b07 (127.3.127.190:38495): Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
I20260812 02:37:43.673843 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:43.668282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60268 (local address 127.3.127.189:43853)
0812 02:37:43.669959 (+ 1677us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:43.669963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:43.669976 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:37:43.670020 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:43.670022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:43.670069 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:43.670169 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:43.670567 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:43.671324 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:43.672902 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:43.673397 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:43.673502 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:43.673662 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:43.673695 (+ 33us) server_negotiation.cc:299] Negotiation successful
0812 02:37:43.673750 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1620,"thread_start_us":68,"threads_started":1}
I20260812 02:37:43.673910 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:43.668209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:43853 (local address 127.0.0.1:60268)
0812 02:37:43.668583 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:43.668599 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:43.668718 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:43.670288 (+ 1570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:43.670291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:43.670310 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:43.670437 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:43.670445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:43.671451 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:43.671455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:43.672758 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:43.672770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:43.673511 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:43.673515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:43.673588 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:43.673604 (+ 16us) client_negotiation.cc:769] Sending connection context
0812 02:37:43.673667 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 02:37:43.673768 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":39,"thread_start_us":77,"threads_started":1}
I20260812 02:37:43.673923 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:43.668911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52544 (local address 127.3.127.188:33229)
0812 02:37:43.669103 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:43.669108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:43.669518 (+ 410us) server_negotiation.cc:407] Connection header received
0812 02:37:43.669716 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:43.669719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:43.669769 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:43.669842 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:43.670359 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:43.671292 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:43.672910 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:43.673349 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:43.673432 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:43.673495 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:43.673651 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 02:37:43.673760 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":137,"thread_start_us":64,"threads_started":1}
I20260812 02:37:43.673843 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:43.668551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:33229 (local address 127.0.0.1:52544)
0812 02:37:43.669268 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:43.669281 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:43.669396 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:43.669973 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:43.669976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:43.669998 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:43.670097 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:43.670103 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:43.671459 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:43.671462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:43.672758 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:43.672770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:43.673454 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:43.673459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:43.673541 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:43.673559 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 02:37:43.673639 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 02:37:43.673711 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":67,"threads_started":1}
I20260812 02:37:43.674568 14699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dd62f6975b334750bba9d180a80ab0cd" is_pre_election: true
I20260812 02:37:43.674568 14991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd62f6975b334750bba9d180a80ab0cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" is_pre_election: true
I20260812 02:37:43.674794 14991 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd62f6975b334750bba9d180a80ab0cd in term 1.
I20260812 02:37:43.674795 14699 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b2ff8a5d080404dbddd48b9a04d9766 in term 1.
I20260812 02:37:43.675141 14956 leader_election.cc:302] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b2ff8a5d080404dbddd48b9a04d9766, dd62f6975b334750bba9d180a80ab0cd; no voters: 05763676674c4a82b5e510ce078c3b07
I20260812 02:37:43.675141 14650 leader_election.cc:302] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b2ff8a5d080404dbddd48b9a04d9766, dd62f6975b334750bba9d180a80ab0cd; no voters: 05763676674c4a82b5e510ce078c3b07
I20260812 02:37:43.675360 15805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 02:37:43.675422 15805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 05763676674c4a82b5e510ce078c3b07)
I20260812 02:37:43.675460 15805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 1 FOLLOWER]: Advancing to term 2
I20260812 02:37:43.675540 15804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 02:37:43.675594 15804 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 05763676674c4a82b5e510ce078c3b07)
I20260812 02:37:43.675639 15804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 1 FOLLOWER]: Advancing to term 2
I20260812 02:37:43.676848 15805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false } }
I20260812 02:37:43.676848 15804 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false } }
I20260812 02:37:43.677261 15804 leader_election.cc:288] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [CANDIDATE]: Term 2 election: Requested vote from peers 05763676674c4a82b5e510ce078c3b07 (127.3.127.190:38495), 0b2ff8a5d080404dbddd48b9a04d9766 (127.3.127.188:33229)
I20260812 02:37:43.677273 15805 leader_election.cc:288] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [CANDIDATE]: Term 2 election: Requested vote from peers 05763676674c4a82b5e510ce078c3b07 (127.3.127.190:38495), dd62f6975b334750bba9d180a80ab0cd (127.3.127.189:43853)
I20260812 02:37:43.677603 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:43.677383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47590)
0812 02:37:43.677476 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:43.677533 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260812 02:37:43.677582 14699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dd62f6975b334750bba9d180a80ab0cd"
I20260812 02:37:43.677728 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:43.677519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47600)
0812 02:37:43.677602 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:43.677665 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 02:37:43.677773 14699 raft_consensus.cc:2379] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b2ff8a5d080404dbddd48b9a04d9766 in current term 2: Already voted for candidate dd62f6975b334750bba9d180a80ab0cd in this term.
I20260812 02:37:43.677858 14991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd62f6975b334750bba9d180a80ab0cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0b2ff8a5d080404dbddd48b9a04d9766"
I20260812 02:37:43.678005 14991 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate dd62f6975b334750bba9d180a80ab0cd in current term 2: Already voted for candidate 0b2ff8a5d080404dbddd48b9a04d9766 in this term.
I20260812 02:37:43.678259 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:43.678018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47610)
0812 02:37:43.678148 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:43.678199 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260812 02:37:43.678367 14954 leader_election.cc:334] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 05763676674c4a82b5e510ce078c3b07 (127.3.127.190:38495): Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
I20260812 02:37:43.678421 14954 leader_election.cc:302] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b2ff8a5d080404dbddd48b9a04d9766; no voters: 05763676674c4a82b5e510ce078c3b07, dd62f6975b334750bba9d180a80ab0cd
I20260812 02:37:43.678550 15805 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 02:37:43.678738 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:43.678520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47626)
0812 02:37:43.678625 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:43.678678 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 02:37:43.678843 14650 leader_election.cc:334] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 05763676674c4a82b5e510ce078c3b07 (127.3.127.190:38495): Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
I20260812 02:37:43.678910 14650 leader_election.cc:302] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd62f6975b334750bba9d180a80ab0cd; no voters: 05763676674c4a82b5e510ce078c3b07, 0b2ff8a5d080404dbddd48b9a04d9766
I20260812 02:37:43.679021 15804 raft_consensus.cc:2726] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 02:37:44.152195 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:44.151358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.130:57657)
0812 02:37:44.151997 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:44.152076 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":120,"threads_started":1}
I20260812 02:37:44.152928 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:44.152559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.131:53321)
0812 02:37:44.152783 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:44.152836 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":73,"threads_started":1}
I20260812 02:37:44.153290 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:44.152819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.129:54927)
0812 02:37:44.153098 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:44.153192 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":87,"threads_started":1}
I20260812 02:37:45.153900 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.153071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.130:47799)
0812 02:37:45.153443 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.153522 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":129,"threads_started":1}
I20260812 02:37:45.154692 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.154232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.129:41811)
0812 02:37:45.154516 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.154579 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":69,"threads_started":1}
I20260812 02:37:45.154774 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.154338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.131:56053)
0812 02:37:45.154615 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.154683 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":66,"threads_started":1}
I20260812 02:37:45.401531 15825 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:37:45.401681 15825 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false } }
I20260812 02:37:45.402083 15825 leader_election.cc:288] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 05763676674c4a82b5e510ce078c3b07 (127.3.127.190:38495), 0b2ff8a5d080404dbddd48b9a04d9766 (127.3.127.188:33229)
I20260812 02:37:45.402751 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.402323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47864)
0812 02:37:45.402563 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.402643 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":71,"threads_started":1}
I20260812 02:37:45.402956 14991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd62f6975b334750bba9d180a80ab0cd" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" is_pre_election: true
I20260812 02:37:45.403120 14991 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd62f6975b334750bba9d180a80ab0cd in term 2.
I20260812 02:37:45.403518 14650 leader_election.cc:302] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b2ff8a5d080404dbddd48b9a04d9766, dd62f6975b334750bba9d180a80ab0cd; no voters:
I20260812 02:37:45.403620 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.403360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47878)
0812 02:37:45.403468 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.403522 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260812 02:37:45.403697 14650 leader_election.cc:334] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 05763676674c4a82b5e510ce078c3b07 (127.3.127.190:38495): Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
I20260812 02:37:45.403738 15825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 02:37:45.403792 15825 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:37:45.403837 15825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 2 FOLLOWER]: Advancing to term 3
I20260812 02:37:45.404939 15825 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false } }
I20260812 02:37:45.405318 15825 leader_election.cc:288] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [CANDIDATE]: Term 3 election: Requested vote from peers 05763676674c4a82b5e510ce078c3b07 (127.3.127.190:38495), 0b2ff8a5d080404dbddd48b9a04d9766 (127.3.127.188:33229)
I20260812 02:37:45.405747 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.405491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47882)
0812 02:37:45.405591 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.405684 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260812 02:37:45.405949 14991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd62f6975b334750bba9d180a80ab0cd" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0b2ff8a5d080404dbddd48b9a04d9766"
I20260812 02:37:45.406083 14991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 2 FOLLOWER]: Advancing to term 3
I20260812 02:37:45.406351 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.406158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47886)
0812 02:37:45.406242 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.406290 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260812 02:37:45.406448 14650 leader_election.cc:334] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 05763676674c4a82b5e510ce078c3b07 (127.3.127.190:38495): Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
I20260812 02:37:45.407218 14991 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd62f6975b334750bba9d180a80ab0cd in term 3.
I20260812 02:37:45.407505 14650 leader_election.cc:302] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b2ff8a5d080404dbddd48b9a04d9766, dd62f6975b334750bba9d180a80ab0cd; no voters: 05763676674c4a82b5e510ce078c3b07
I20260812 02:37:45.407678 15825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 3 FOLLOWER]: Leader election won for term 3
I20260812 02:37:45.407891 15825 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [term 3 LEADER]: Becoming Leader. State: Replica: dd62f6975b334750bba9d180a80ab0cd, State: Running, Role: LEADER
I20260812 02:37:45.408145 15825 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false } }
I20260812 02:37:45.409708 15829 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd62f6975b334750bba9d180a80ab0cd. Latest consensus state: current_term: 3 leader_uuid: "dd62f6975b334750bba9d180a80ab0cd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false } } }
I20260812 02:37:45.409822 15829 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:45.410140 15833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:37:45.412546 15833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:37:45.412915 15833 catalog_manager.cc:1259] Loaded cluster ID: 1edca9d490854f83b1c360530e4749c0
I20260812 02:37:45.412962 15833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:37:45.413532 15833 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:37:45.413884 15833 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: Loaded TSK: 0
I20260812 02:37:45.414533 15833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:37:45.430613 14673 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:37:45.433713 14673 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:37:45.436905 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.436623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47908)
0812 02:37:45.436772 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.436833 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260812 02:37:45.437284 14991 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [term 3 FOLLOWER]: Refusing update from remote peer dd62f6975b334750bba9d180a80ab0cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260812 02:37:45.437674 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.437427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47922)
0812 02:37:45.437540 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.437604 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260812 02:37:45.437857 14650 consensus_peers.cc:487] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd -> Peer 05763676674c4a82b5e510ce078c3b07 (127.3.127.190:38495): Couldn't send request to peer 05763676674c4a82b5e510ce078c3b07. Status: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 02:37:45.437865 15825 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260812 02:37:45.439567 15836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd62f6975b334750bba9d180a80ab0cd. Latest consensus state: current_term: 3 leader_uuid: "dd62f6975b334750bba9d180a80ab0cd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false } } }
I20260812 02:37:45.439689 15836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:37:45.440897 15829 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "dd62f6975b334750bba9d180a80ab0cd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false } } }
I20260812 02:37:45.441027 15829 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:45.441375 15836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "dd62f6975b334750bba9d180a80ab0cd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05763676674c4a82b5e510ce078c3b07" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38495 } } peers { permanent_uuid: "dd62f6975b334750bba9d180a80ab0cd" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 43853 } attrs { promote: false } } peers { permanent_uuid: "0b2ff8a5d080404dbddd48b9a04d9766" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 33229 } attrs { promote: false } } }
I20260812 02:37:45.441497 15836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b2ff8a5d080404dbddd48b9a04d9766 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:37:45.452018 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.447923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:37785 (local address 127.0.0.1:60924)
0812 02:37:45.448500 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.448513 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:45.448584 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:45.449010 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:45.449013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:45.449031 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:45.449124 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:45.449130 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:45.449892 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:45.449895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:45.450852 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:45.450861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:45.451599 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:45.451602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:45.451698 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:45.451723 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 02:37:45.451790 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 02:37:45.451868 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":134,"threads_started":1}
I20260812 02:37:45.452018 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.448326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60924 (local address 127.3.127.129:37785)
0812 02:37:45.448793 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:45.448798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:45.448813 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:37:45.448865 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:45.448868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:45.448914 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:45.448997 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:45.449250 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:45.449785 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:45.450993 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:45.451478 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:45.451581 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:45.451654 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:45.451814 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 02:37:45.451899 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":67,"threads_started":1}
I20260812 02:37:45.452607 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.447631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:40167 (local address 127.0.0.1:46160)
0812 02:37:45.448193 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.448208 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:45.448312 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:45.448811 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:45.448814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:45.448834 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:45.448946 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:45.448952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:45.450090 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:45.450093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:45.451269 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:45.451281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:45.452244 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:45.452248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:45.452334 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:45.452356 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 02:37:45.452419 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 02:37:45.452485 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496}
I20260812 02:37:45.452637 15165 tablet_service.cc:1467] Processing CreateTablet for tablet 563a3729dcff4ceca99d64c679a07c5b (DEFAULT_TABLE table=test-table [id=0688a383d9dc4d679864852e4836ba80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:45.452962 15165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 563a3729dcff4ceca99d64c679a07c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:45.452994 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.447765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.3.127.131:40167)
0812 02:37:45.448040 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:45.448044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:45.448292 (+ 248us) server_negotiation.cc:407] Connection header received
0812 02:37:45.448403 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:45.448406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:45.448457 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:45.448539 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:45.449077 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:45.449952 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:45.451414 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:45.452019 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:45.452116 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:45.452186 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:45.452784 (+ 598us) server_negotiation.cc:299] Negotiation successful
0812 02:37:45.452870 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20260812 02:37:45.453645 15584 tablet_service.cc:1467] Processing CreateTablet for tablet 563a3729dcff4ceca99d64c679a07c5b (DEFAULT_TABLE table=test-table [id=0688a383d9dc4d679864852e4836ba80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:45.453914 15584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 563a3729dcff4ceca99d64c679a07c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:45.455142 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.448593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46324 (local address 127.3.127.130:34357)
0812 02:37:45.448926 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:45.448930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:45.451688 (+ 2758us) server_negotiation.cc:407] Connection header received
0812 02:37:45.451750 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:45.451753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:45.451802 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:45.451872 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:45.452240 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:45.452959 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:45.454173 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:45.454596 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:45.454703 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:45.454767 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:45.454919 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 02:37:45.455001 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":111,"threads_started":1}
I20260812 02:37:45.455139 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.448250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:34357 (local address 127.0.0.1:46324)
0812 02:37:45.448605 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.448619 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:45.451601 (+ 2982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:45.451980 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:45.451984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:45.452001 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:45.452099 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:45.452105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:45.453092 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:45.453095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:45.454051 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:45.454061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:45.454717 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:45.454720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:45.454806 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:45.454828 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 02:37:45.454901 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 02:37:45.455012 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":99,"threads_started":1}
I20260812 02:37:45.455744 15856 tablet_bootstrap.cc:492] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434: Bootstrap starting.
I20260812 02:37:45.456418 15857 tablet_bootstrap.cc:492] T 563a3729dcff4ceca99d64c679a07c5b P 6f0f40aab58f4476a0c404dfce26053c: Bootstrap starting.
I20260812 02:37:45.456696 15856 tablet_bootstrap.cc:654] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:45.457427 15857 tablet_bootstrap.cc:654] T 563a3729dcff4ceca99d64c679a07c5b P 6f0f40aab58f4476a0c404dfce26053c: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:45.457430 15382 tablet_service.cc:1467] Processing CreateTablet for tablet 563a3729dcff4ceca99d64c679a07c5b (DEFAULT_TABLE table=test-table [id=0688a383d9dc4d679864852e4836ba80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:45.457705 15382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 563a3729dcff4ceca99d64c679a07c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:45.458015 15856 tablet_bootstrap.cc:492] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434: No bootstrap required, opened a new log
I20260812 02:37:45.458122 15856 ts_tablet_manager.cc:1397] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 02:37:45.458540 15856 raft_consensus.cc:348] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } } peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } }
I20260812 02:37:45.458621 15857 tablet_bootstrap.cc:492] T 563a3729dcff4ceca99d64c679a07c5b P 6f0f40aab58f4476a0c404dfce26053c: No bootstrap required, opened a new log
I20260812 02:37:45.458643 15856 raft_consensus.cc:374] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:45.458676 15856 raft_consensus.cc:729] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 430e867652ed41ff8cda09459a710434, State: Initialized, Role: FOLLOWER
I20260812 02:37:45.458698 15857 ts_tablet_manager.cc:1397] T 563a3729dcff4ceca99d64c679a07c5b P 6f0f40aab58f4476a0c404dfce26053c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 02:37:45.458832 15856 consensus_queue.cc:260] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } } peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } }
I20260812 02:37:45.459221 15856 ts_tablet_manager.cc:1428] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:37:45.459215 15857 raft_consensus.cc:348] T 563a3729dcff4ceca99d64c679a07c5b P 6f0f40aab58f4476a0c404dfce26053c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } } peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } }
I20260812 02:37:45.459395 15857 raft_consensus.cc:374] T 563a3729dcff4ceca99d64c679a07c5b P 6f0f40aab58f4476a0c404dfce26053c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:45.459470 15857 raft_consensus.cc:729] T 563a3729dcff4ceca99d64c679a07c5b P 6f0f40aab58f4476a0c404dfce26053c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f0f40aab58f4476a0c404dfce26053c, State: Initialized, Role: FOLLOWER
I20260812 02:37:45.459666 15857 consensus_queue.cc:260] T 563a3729dcff4ceca99d64c679a07c5b P 6f0f40aab58f4476a0c404dfce26053c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } } peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } }
I20260812 02:37:45.459910 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.459662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.129:51657)
0812 02:37:45.459774 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.459836 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 02:37:45.460101 15857 ts_tablet_manager.cc:1428] T 563a3729dcff4ceca99d64c679a07c5b P 6f0f40aab58f4476a0c404dfce26053c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:37:45.460289 15865 tablet_bootstrap.cc:492] T 563a3729dcff4ceca99d64c679a07c5b P 3737fe1f65fa40a484cc4f7dc3e030ba: Bootstrap starting.
I20260812 02:37:45.460855 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.460583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.131:43863)
0812 02:37:45.460737 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.460788 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260812 02:37:45.461259 15865 tablet_bootstrap.cc:654] T 563a3729dcff4ceca99d64c679a07c5b P 3737fe1f65fa40a484cc4f7dc3e030ba: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:45.461733 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.461464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.129:33959)
0812 02:37:45.461614 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.461666 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260812 02:37:45.462853 15267 heartbeater.cc:503] Master 127.3.127.189:43853 was elected leader, sending a full tablet report...
I20260812 02:37:45.464685 15691 heartbeater.cc:503] Master 127.3.127.189:43853 was elected leader, sending a full tablet report...
I20260812 02:37:45.464815 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.464058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.131:43225)
0812 02:37:45.464644 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.464693 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525}
I20260812 02:37:45.464946 15865 tablet_bootstrap.cc:492] T 563a3729dcff4ceca99d64c679a07c5b P 3737fe1f65fa40a484cc4f7dc3e030ba: No bootstrap required, opened a new log
I20260812 02:37:45.465041 15865 ts_tablet_manager.cc:1397] T 563a3729dcff4ceca99d64c679a07c5b P 3737fe1f65fa40a484cc4f7dc3e030ba: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260812 02:37:45.465585 15865 raft_consensus.cc:348] T 563a3729dcff4ceca99d64c679a07c5b P 3737fe1f65fa40a484cc4f7dc3e030ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } } peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } }
I20260812 02:37:45.465728 15865 raft_consensus.cc:374] T 563a3729dcff4ceca99d64c679a07c5b P 3737fe1f65fa40a484cc4f7dc3e030ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:45.465765 15865 raft_consensus.cc:729] T 563a3729dcff4ceca99d64c679a07c5b P 3737fe1f65fa40a484cc4f7dc3e030ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3737fe1f65fa40a484cc4f7dc3e030ba, State: Initialized, Role: FOLLOWER
I20260812 02:37:45.465900 15865 consensus_queue.cc:260] T 563a3729dcff4ceca99d64c679a07c5b P 3737fe1f65fa40a484cc4f7dc3e030ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } } peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } }
I20260812 02:37:45.466305 15865 ts_tablet_manager.cc:1428] T 563a3729dcff4ceca99d64c679a07c5b P 3737fe1f65fa40a484cc4f7dc3e030ba: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260812 02:37:45.466998 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.466614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.130:33539)
0812 02:37:45.466865 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.466924 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260812 02:37:45.468027 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.467762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.130:51989)
0812 02:37:45.467900 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.467950 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260812 02:37:45.468163 15482 heartbeater.cc:503] Master 127.3.127.189:43853 was elected leader, sending a full tablet report...
W20260812 02:37:45.494552 15484 tablet.cc:2369] T 563a3729dcff4ceca99d64c679a07c5b P 3737fe1f65fa40a484cc4f7dc3e030ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:37:45.703919 15692 tablet.cc:2369] T 563a3729dcff4ceca99d64c679a07c5b P 6f0f40aab58f4476a0c404dfce26053c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:37:45.705760 15268 tablet.cc:2369] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:37:45.870741 15862 raft_consensus.cc:482] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:37:45.870918 15862 raft_consensus.cc:504] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } } peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } }
I20260812 02:37:45.871389 15862 leader_election.cc:288] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f0f40aab58f4476a0c404dfce26053c (127.3.127.131:40167), 3737fe1f65fa40a484cc4f7dc3e030ba (127.3.127.130:34357)
I20260812 02:37:45.871925 15619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563a3729dcff4ceca99d64c679a07c5b" candidate_uuid: "430e867652ed41ff8cda09459a710434" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f0f40aab58f4476a0c404dfce26053c" is_pre_election: true
I20260812 02:37:45.871994 15408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563a3729dcff4ceca99d64c679a07c5b" candidate_uuid: "430e867652ed41ff8cda09459a710434" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" is_pre_election: true
I20260812 02:37:45.872088 15619 raft_consensus.cc:2454] T 563a3729dcff4ceca99d64c679a07c5b P 6f0f40aab58f4476a0c404dfce26053c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 430e867652ed41ff8cda09459a710434 in term 0.
I20260812 02:37:45.872184 15408 raft_consensus.cc:2454] T 563a3729dcff4ceca99d64c679a07c5b P 3737fe1f65fa40a484cc4f7dc3e030ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 430e867652ed41ff8cda09459a710434 in term 0.
I20260812 02:37:45.872455 15089 leader_election.cc:302] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 430e867652ed41ff8cda09459a710434, 6f0f40aab58f4476a0c404dfce26053c; no voters:
I20260812 02:37:45.872678 15862 raft_consensus.cc:2781] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:37:45.872742 15862 raft_consensus.cc:482] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:37:45.872788 15862 raft_consensus.cc:3037] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:45.875801 15862 raft_consensus.cc:504] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } } peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } }
I20260812 02:37:45.876250 15862 leader_election.cc:288] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 [CANDIDATE]: Term 1 election: Requested vote from peers 6f0f40aab58f4476a0c404dfce26053c (127.3.127.131:40167), 3737fe1f65fa40a484cc4f7dc3e030ba (127.3.127.130:34357)
I20260812 02:37:45.876737 15619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563a3729dcff4ceca99d64c679a07c5b" candidate_uuid: "430e867652ed41ff8cda09459a710434" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f0f40aab58f4476a0c404dfce26053c"
I20260812 02:37:45.876780 15408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563a3729dcff4ceca99d64c679a07c5b" candidate_uuid: "430e867652ed41ff8cda09459a710434" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba"
I20260812 02:37:45.876894 15619 raft_consensus.cc:3037] T 563a3729dcff4ceca99d64c679a07c5b P 6f0f40aab58f4476a0c404dfce26053c [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:45.876900 15408 raft_consensus.cc:3037] T 563a3729dcff4ceca99d64c679a07c5b P 3737fe1f65fa40a484cc4f7dc3e030ba [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:45.878132 15408 raft_consensus.cc:2454] T 563a3729dcff4ceca99d64c679a07c5b P 3737fe1f65fa40a484cc4f7dc3e030ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 430e867652ed41ff8cda09459a710434 in term 1.
I20260812 02:37:45.878152 15619 raft_consensus.cc:2454] T 563a3729dcff4ceca99d64c679a07c5b P 6f0f40aab58f4476a0c404dfce26053c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 430e867652ed41ff8cda09459a710434 in term 1.
I20260812 02:37:45.878497 15089 leader_election.cc:302] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 430e867652ed41ff8cda09459a710434, 6f0f40aab58f4476a0c404dfce26053c; no voters:
I20260812 02:37:45.878702 15862 raft_consensus.cc:2781] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:45.878935 15862 raft_consensus.cc:686] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 [term 1 LEADER]: Becoming Leader. State: Replica: 430e867652ed41ff8cda09459a710434, State: Running, Role: LEADER
I20260812 02:37:45.879106 15862 consensus_queue.cc:237] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } } peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } }
I20260812 02:37:45.879817 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.879505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.129:47961)
0812 02:37:45.879655 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.879728 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 02:37:45.880897 14673 catalog_manager.cc:5495] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 reported cstate change: term changed from 0 to 1, leader changed from <none> to 430e867652ed41ff8cda09459a710434 (127.3.127.129). New cstate: current_term: 1 leader_uuid: "430e867652ed41ff8cda09459a710434" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "430e867652ed41ff8cda09459a710434" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 37785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 } health_report { overall_health: UNKNOWN } } }
I20260812 02:37:45.887959 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.887701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47924)
0812 02:37:45.887819 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.887880 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260812 02:37:45.888656 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.888421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.0.0.1:47940)
0812 02:37:45.888517 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.888567 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260812 02:37:45.915584 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.912427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60934 (local address 127.3.127.129:37785)
0812 02:37:45.912611 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:45.912617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:45.912633 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:37:45.912688 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:45.912690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:45.912735 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:45.912799 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:45.913056 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:45.913648 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:45.914491 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:45.914769 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:45.914863 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:45.915110 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:45.915190 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:45.915248 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:45.915360 (+ 112us) server_negotiation.cc:299] Negotiation successful
0812 02:37:45.915422 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 02:37:45.917855 15619 raft_consensus.cc:1261] T 563a3729dcff4ceca99d64c679a07c5b P 6f0f40aab58f4476a0c404dfce26053c [term 1 FOLLOWER]: Refusing update from remote peer 430e867652ed41ff8cda09459a710434: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:45.917853 15408 raft_consensus.cc:1261] T 563a3729dcff4ceca99d64c679a07c5b P 3737fe1f65fa40a484cc4f7dc3e030ba [term 1 FOLLOWER]: Refusing update from remote peer 430e867652ed41ff8cda09459a710434: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:45.918349 15862 consensus_queue.cc:1035] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f0f40aab58f4476a0c404dfce26053c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 40167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:45.918526 15869 consensus_queue.cc:1035] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3737fe1f65fa40a484cc4f7dc3e030ba" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:45.920431 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.919824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.131:44623)
0812 02:37:45.920280 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.920351 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380}
I20260812 02:37:45.921445 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.920979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.130:60977)
0812 02:37:45.921331 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.921384 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283}
I20260812 02:37:45.923588 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.923353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.129:49517)
0812 02:37:45.923451 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.923506 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 02:37:45.925009 15876 mvcc.cc:205] Tried to move back new op lower bound from 7317513281195065344 to 7317513281042362368. Current Snapshot: MvccSnapshot[applied={T|T < 7317513281195065344}]
I20260812 02:37:45.927651 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.927302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.130:54957)
0812 02:37:45.927503 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.927567 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260812 02:37:45.927649 14684 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:45.927812 14684 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:45.928668 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.928232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.129:49915)
0812 02:37:45.928528 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.928587 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221}
I20260812 02:37:45.930357 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.929349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38495 (local address 127.3.127.131:33689)
0812 02:37:45.930219 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.930279 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:38495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":799}
I20260812 02:37:45.932930 14684 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd62f6975b334750bba9d180a80ab0cd: Sending DeleteTablet for 3 replicas of tablet 563a3729dcff4ceca99d64c679a07c5b
I20260812 02:37:45.933718 15584 tablet_service.cc:1514] Processing DeleteTablet for tablet 563a3729dcff4ceca99d64c679a07c5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:45 UTC) from {username='oryx', principal='oryx/127.3.127.189@KRBTEST.COM'} at 127.0.0.1:46160
I20260812 02:37:45.933837 15165 tablet_service.cc:1514] Processing DeleteTablet for tablet 563a3729dcff4ceca99d64c679a07c5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:45 UTC) from {username='oryx', principal='oryx/127.3.127.189@KRBTEST.COM'} at 127.0.0.1:60924
I20260812 02:37:45.934147 15382 tablet_service.cc:1514] Processing DeleteTablet for tablet 563a3729dcff4ceca99d64c679a07c5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:45 UTC) from {username='oryx', principal='oryx/127.3.127.189@KRBTEST.COM'} at 127.0.0.1:46324
I20260812 02:37:45.934257 15887 tablet_replica.cc:330] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434: stopping tablet replica
I20260812 02:37:45.934383 15887 raft_consensus.cc:2229] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:37:45.934470 15886 tablet_replica.cc:330] T 563a3729dcff4ceca99d64c679a07c5b P 6f0f40aab58f4476a0c404dfce26053c: stopping tablet replica
I20260812 02:37:45.934581 15887 raft_consensus.cc:2258] T 563a3729dcff4ceca99d64c679a07c5b P 430e867652ed41ff8cda09459a710434 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:45.934600 15886 raft_consensus.cc:2229] T 563a3729dcff4ceca99d64c679a07c5b P 6f0f40aab58f4476a0c404dfce26053c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:45.934707 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 14918
I20260812 02:37:45.934729 15886 raft_consensus.cc:2258] T 563a3729dcff4ceca99d64c679a07c5b P 6f0f40aab58f4476a0c404dfce26053c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:45.944231 15889 tablet_replica.cc:330] T 563a3729dcff4ceca99d64c679a07c5b P 3737fe1f65fa40a484cc4f7dc3e030ba: stopping tablet replica
I20260812 02:37:45.944408 15889 raft_consensus.cc:2229] T 563a3729dcff4ceca99d64c679a07c5b P 3737fe1f65fa40a484cc4f7dc3e030ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:45.944483 15886 ts_tablet_manager.cc:1905] T 563a3729dcff4ceca99d64c679a07c5b P 6f0f40aab58f4476a0c404dfce26053c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:45.944656 15889 raft_consensus.cc:2258] T 563a3729dcff4ceca99d64c679a07c5b P 3737fe1f65fa40a484cc4f7dc3e030ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:45.945482 15889 ts_tablet_manager.cc:1905] T 563a3729dcff4ceca99d64c679a07c5b P 3737fe1f65fa40a484cc4f7dc3e030ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:45.947395 15886 ts_tablet_manager.cc:1918] T 563a3729dcff4ceca99d64c679a07c5b P 6f0f40aab58f4476a0c404dfce26053c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:37:45.947491 15886 log.cc:1196] T 563a3729dcff4ceca99d64c679a07c5b P 6f0f40aab58f4476a0c404dfce26053c: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/563a3729dcff4ceca99d64c679a07c5b
I20260812 02:37:45.947786 15886 ts_tablet_manager.cc:1939] T 563a3729dcff4ceca99d64c679a07c5b P 6f0f40aab58f4476a0c404dfce26053c: Deleting consensus metadata
I20260812 02:37:45.948019 15889 ts_tablet_manager.cc:1918] T 563a3729dcff4ceca99d64c679a07c5b P 3737fe1f65fa40a484cc4f7dc3e030ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:37:45.948110 15889 log.cc:1196] T 563a3729dcff4ceca99d64c679a07c5b P 3737fe1f65fa40a484cc4f7dc3e030ba: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/563a3729dcff4ceca99d64c679a07c5b
I20260812 02:37:45.948369 15889 ts_tablet_manager.cc:1939] T 563a3729dcff4ceca99d64c679a07c5b P 3737fe1f65fa40a484cc4f7dc3e030ba: Deleting consensus metadata
I20260812 02:37:45.948762 14650 catalog_manager.cc:4841] TS 6f0f40aab58f4476a0c404dfce26053c (127.3.127.131:40167): tablet 563a3729dcff4ceca99d64c679a07c5b (table test-table [id=0688a383d9dc4d679864852e4836ba80]) successfully deleted
I20260812 02:37:45.949081 14650 catalog_manager.cc:4841] TS 3737fe1f65fa40a484cc4f7dc3e030ba (127.3.127.130:34357): tablet 563a3729dcff4ceca99d64c679a07c5b (table test-table [id=0688a383d9dc4d679864852e4836ba80]) successfully deleted
W20260812 02:37:45.954962 14656 connection.cc:535] server connection from 127.3.127.129:33997 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 02:37:45.955036 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 15274
I20260812 02:37:45.955534 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:45.955228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:37785 (local address 127.0.0.1:60936)
0812 02:37:45.955406 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:45.955450 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.129:37785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260812 02:37:45.963435 14648 catalog_manager.cc:4584] TS 430e867652ed41ff8cda09459a710434 (127.3.127.129:37785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 563a3729dcff4ceca99d64c679a07c5b: Network error: Client connection negotiation failed: client connection to 127.3.127.129:37785: connect: Connection refused (error 111)
I20260812 02:37:45.973289 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 15488
I20260812 02:37:45.988075 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 14459
I20260812 02:37:46.003281 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 14625
2026-08-12T02:37:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7837 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:37:46 dist-test-slave-rmhg krb5kdc[15894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:37:46 dist-test-slave-rmhg krb5kdc[15894](info): set up 2 sockets
Aug 12 02:37:46 dist-test-slave-rmhg krb5kdc[15894](info): commencing operation
krb5kdc: starting...
W20260812 02:37:47.201674 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:37:47 dist-test-slave-rmhg krb5kdc[15894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:37:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:37:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.190.keytab.
Entry for principal oryx/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.190.keytab.
I20260812 02:37:47.341596 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:40727
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.190.keytab
--principal=oryx/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:36191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:40727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:47.463666 15920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:47.463980 15920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:47.464023 15920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:47.468184 15920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:47.468263 15920 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 02:37:47.468281 15920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:47.468297 15920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:47.468312 15920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:47.468325 15920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:37:47.474164 15920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:36191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:40727
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.190.keytab
--principal=oryx/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:40727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:47.475339 15920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:47.476400 15920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:47.485751 15934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:47.485828 15931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:47.485801 15929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:47.485950 15920 server_base.cc:1034] running on GCE node
I20260812 02:37:47.486686 15920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:47.487994 15920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:47.489190 15920 hybrid_clock.cc:648] HybridClock initialized: now 1786502267489166 us; error 35 us; skew 500 ppm
Aug 12 02:37:47 dist-test-slave-rmhg krb5kdc[15894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502267, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:47.492388 15920 init.cc:377] Logged in from keytab as oryx/127.3.127.190@KRBTEST.COM (short username oryx)
I20260812 02:37:47.493538 15920 webserver.cc:460] Webserver started at http://127.3.127.190:46321/ using document root <none> and password file <none>
I20260812 02:37:47.494146 15920 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:47.494215 15920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:47.494423 15920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:47.496655 15920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "fe2c7796e32049f2ba7710104f5520ea"
format_stamp: "Formatted at 2026-08-12 02:37:47 on dist-test-slave-rmhg"
server_key: "79e8cd5cfac2c5918ff49a3a8d61cabd"
server_key_iv: "2b803b9af53ca412d8a31eda6651b95d"
server_key_version: "encryptionkey@0"
I20260812 02:37:47.497305 15920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "fe2c7796e32049f2ba7710104f5520ea"
format_stamp: "Formatted at 2026-08-12 02:37:47 on dist-test-slave-rmhg"
server_key: "79e8cd5cfac2c5918ff49a3a8d61cabd"
server_key_iv: "2b803b9af53ca412d8a31eda6651b95d"
server_key_version: "encryptionkey@0"
I20260812 02:37:47.501533 15920 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:37:47.504477 15943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:47.506109 15920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:47.506273 15920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "fe2c7796e32049f2ba7710104f5520ea"
format_stamp: "Formatted at 2026-08-12 02:37:47 on dist-test-slave-rmhg"
server_key: "79e8cd5cfac2c5918ff49a3a8d61cabd"
server_key_iv: "2b803b9af53ca412d8a31eda6651b95d"
server_key_version: "encryptionkey@0"
I20260812 02:37:47.506398 15920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:47.528689 15920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:47.532172 15920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:47.532377 15920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:47.540658 15920 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:40727
I20260812 02:37:47.540683 16043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:40727 every 8 connection(s)
I20260812 02:37:47.541738 15920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:37:47.544884 16044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:47.550196 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 15920
I20260812 02:37:47.550355 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:37:47.550732 3582 external_mini_cluster.cc:1428] Setting key 53c2e776d0e8efbba5deb010a74be097
I20260812 02:37:47.551281 16044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea: Bootstrap starting.
I20260812 02:37:47.553998 16044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:47.554826 16044 log.cc:824] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:47.557021 16044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea: No bootstrap required, opened a new log
Aug 12 02:37:47 dist-test-slave-rmhg krb5kdc[15894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM
I20260812 02:37:47.561041 16044 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2c7796e32049f2ba7710104f5520ea" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40727 } }
I20260812 02:37:47.561352 16044 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:47.561415 16044 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe2c7796e32049f2ba7710104f5520ea, State: Initialized, Role: FOLLOWER
I20260812 02:37:47.561914 16044 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2c7796e32049f2ba7710104f5520ea" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40727 } }
I20260812 02:37:47.562078 16044 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:37:47.562155 16044 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:37:47.562260 16044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:47.563215 16044 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2c7796e32049f2ba7710104f5520ea" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40727 } }
I20260812 02:37:47.563622 16044 leader_election.cc:302] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe2c7796e32049f2ba7710104f5520ea; no voters:
I20260812 02:37:47.564013 16044 leader_election.cc:288] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:37:47.564134 16051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:47.564411 16051 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea [term 1 LEADER]: Becoming Leader. State: Replica: fe2c7796e32049f2ba7710104f5520ea, State: Running, Role: LEADER
I20260812 02:37:47.564862 16051 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2c7796e32049f2ba7710104f5520ea" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40727 } }
I20260812 02:37:47.565506 16044 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:37:47.566249 16047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:47.552263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49612 (local address 127.3.127.190:40727)
0812 02:37:47.552718 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:47.552735 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:47.552774 (+ 39us) server_negotiation.cc:407] Connection header received
0812 02:37:47.553500 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:47.553524 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:47.553924 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:47.554302 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:47.555167 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:47.556243 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:47.557200 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:47.557707 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:47.557866 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:47.560631 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:47.560636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:47.560652 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:47.560684 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:47.562517 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:47.562978 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:47.562985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:47.562990 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:47.563119 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:47.563409 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:47.563414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:47.563415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:47.564063 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:47.564272 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:47.564787 (+ 515us) server_negotiation.cc:299] Negotiation successful
0812 02:37:47.565131 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":125,"threads_started":1}
I20260812 02:37:47.566823 16052 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe2c7796e32049f2ba7710104f5520ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2c7796e32049f2ba7710104f5520ea" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40727 } } }
I20260812 02:37:47.566949 16052 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:47.567174 16052 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe2c7796e32049f2ba7710104f5520ea. Latest consensus state: current_term: 1 leader_uuid: "fe2c7796e32049f2ba7710104f5520ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2c7796e32049f2ba7710104f5520ea" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40727 } } }
I20260812 02:37:47.567245 16052 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:47.567643 16071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:37:47.571369 16071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:37:47.577603 16071 catalog_manager.cc:1347] Generated new cluster ID: 4038457b0ba14f06a90d0f6fd6b50a16
I20260812 02:37:47.577704 16071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:37:47.596611 16071 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:37:47.598073 16071 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:37:47.604018 16071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea: Generated new TSK 0
I20260812 02:37:47.604796 16071 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.129.keytab.
Entry for principal oryx/127.3.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.129.keytab.
I20260812 02:37:47.641884 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:40727
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.129.keytab
--principal=oryx/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:36191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:47.763839 16090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:47.764139 16090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:47.764191 16090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:47.768597 16090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:47.768684 16090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:47.768726 16090 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:37:47.768782 16090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:37:47.774549 16090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:36191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.129.keytab
--principal=oryx/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.127.190:40727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:47.775702 16090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:47.776754 16090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:47.787607 16107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:47.787612 16097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:47.787845 16090 server_base.cc:1034] running on GCE node
W20260812 02:37:47.787607 16099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:47.788328 16090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:47.789074 16090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:47.790256 16090 hybrid_clock.cc:648] HybridClock initialized: now 1786502267790230 us; error 43 us; skew 500 ppm
Aug 12 02:37:47 dist-test-slave-rmhg krb5kdc[15894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502267, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:47.793406 16090 init.cc:377] Logged in from keytab as oryx/127.3.127.129@KRBTEST.COM (short username oryx)
I20260812 02:37:47.794487 16090 webserver.cc:460] Webserver started at http://127.3.127.129:37789/ using document root <none> and password file <none>
I20260812 02:37:47.795101 16090 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:47.795173 16090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:47.795382 16090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:47.797631 16090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "4cf33640abc844fe8199f1bd5d41a61e"
format_stamp: "Formatted at 2026-08-12 02:37:47 on dist-test-slave-rmhg"
server_key: "eb1bc0fd0d1cab406fd3a452e618c56f"
server_key_iv: "2fecba1e06d6371a642d72749f4ba13a"
server_key_version: "encryptionkey@0"
I20260812 02:37:47.798218 16090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "4cf33640abc844fe8199f1bd5d41a61e"
format_stamp: "Formatted at 2026-08-12 02:37:47 on dist-test-slave-rmhg"
server_key: "eb1bc0fd0d1cab406fd3a452e618c56f"
server_key_iv: "2fecba1e06d6371a642d72749f4ba13a"
server_key_version: "encryptionkey@0"
I20260812 02:37:47.802301 16090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:37:47.805040 16123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:47.806516 16090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 02:37:47.806736 16090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "4cf33640abc844fe8199f1bd5d41a61e"
format_stamp: "Formatted at 2026-08-12 02:37:47 on dist-test-slave-rmhg"
server_key: "eb1bc0fd0d1cab406fd3a452e618c56f"
server_key_iv: "2fecba1e06d6371a642d72749f4ba13a"
server_key_version: "encryptionkey@0"
I20260812 02:37:47.806862 16090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:47.831241 16090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:47.834517 16090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:47.834724 16090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:47.836287 16090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:47.836341 16090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:47.836398 16090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:47.836428 16090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:37:47 dist-test-slave-rmhg krb5kdc[15894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502267, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM
I20260812 02:37:47.847679 16090 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:40483
I20260812 02:37:47.848083 16314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:40483 every 8 connection(s)
I20260812 02:37:47.848799 16090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:37:47.850396 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 16090
I20260812 02:37:47.850544 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:37:47.850889 3582 external_mini_cluster.cc:1428] Setting key c131ead72736816a45f98e78cc32ef45
I20260812 02:37:47.851372 16047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:47.837294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:49487 (local address 127.3.127.190:40727)
0812 02:37:47.837419 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:47.837423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:47.838600 (+ 1177us) server_negotiation.cc:407] Connection header received
0812 02:37:47.839737 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:47.839742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:47.839811 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:47.839872 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:47.841622 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:47.842206 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:47.843551 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:47.843808 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:47.843906 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:47.847090 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:47.847094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:47.847097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:47.847133 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:47.848825 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:47.849517 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:47.849522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:47.849524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:47.849639 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:47.850116 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:47.850119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:47.850120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:47.850328 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:47.850400 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:47.851077 (+ 677us) server_negotiation.cc:299] Negotiation successful
0812 02:37:47.851225 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 02:37:47.852408 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:47.837869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:40727 (local address 127.3.127.129:49487)
0812 02:37:47.838423 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:47.838472 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:47.839442 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:47.840322 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:47.840329 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:47.840729 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:47.841443 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:47.841462 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:47.842350 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:47.842353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:47.843406 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:47.843421 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:47.843998 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:47.844003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:47.844203 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:47.844859 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:47.844882 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:47.846883 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:47.848975 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:47.848981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:47.848997 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:47.849392 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:47.849889 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:47.849892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:47.849893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:47.850035 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:47.850427 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:47.850435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:47.850784 (+ 349us) client_negotiation.cc:769] Sending connection context
0812 02:37:47.850997 (+ 213us) client_negotiation.cc:240] Negotiation successful
0812 02:37:47.851385 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":153,"threads_started":1}
I20260812 02:37:47.854844 16316 heartbeater.cc:347] Connected to a master server at 127.3.127.190:40727
I20260812 02:37:47.855067 16316 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:47.855692 16316 heartbeater.cc:511] Master 127.3.127.190:40727 requested a full tablet report, sending...
I20260812 02:37:47.857517 15973 ts_manager.cc:194] Registered new tserver with Master: 4cf33640abc844fe8199f1bd5d41a61e (127.3.127.129:40483)
I20260812 02:37:47.858954 15973 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:49487
WARNING: no policy specified for oryx/127.3.127.130@KRBTEST.COM; defaulting to no policy
I20260812 02:37:47.864295 16047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:47.856695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:46207 (local address 127.3.127.190:40727)
0812 02:37:47.856802 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:47.856807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:47.856819 (+ 12us) server_negotiation.cc:407] Connection header received
0812 02:37:47.856857 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:47.856860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:47.856901 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:47.856978 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:47.857560 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:47.858364 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:47.859485 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:47.859822 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:47.859961 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:47.861354 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:47.861359 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:47.861361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:47.861390 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:47.862879 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:47.863277 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:47.863280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:47.863281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:47.863369 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:47.863608 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:47.863609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:47.863611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:47.863787 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:47.863862 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:47.864065 (+ 203us) server_negotiation.cc:299] Negotiation successful
0812 02:37:47.864171 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 02:37:47.864323 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:47.856332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:40727 (local address 127.3.127.129:46207)
0812 02:37:47.856637 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:47.856652 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:47.856751 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:47.857010 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:47.857013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:47.857279 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:47.857422 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:47.857430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:47.858508 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:47.858513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:47.859351 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:47.859363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:47.859975 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:47.859978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:47.860086 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:47.860696 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:47.860711 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:47.861200 (+ 489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:47.862981 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:47.862984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:47.862985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:47.863196 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:47.863448 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:47.863452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:47.863454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:47.863540 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:47.863853 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:47.863856 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:47.863936 (+ 80us) client_negotiation.cc:769] Sending connection context
0812 02:37:47.864012 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 02:37:47.864134 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.127.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.130.keytab.
Entry for principal oryx/127.3.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.130.keytab.
I20260812 02:37:47.881908 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:40727
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.130.keytab
--principal=oryx/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:36191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:48.001327 16330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:48.001644 16330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:48.001713 16330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:48.005934 16330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:48.006019 16330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:48.006078 16330 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:37:48.006129 16330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:37:48.011947 16330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:36191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.130.keytab
--principal=oryx/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.127.190:40727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:48.013202 16330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:48.014335 16330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:48.024727 16341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:48.024683 16344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:48.024688 16342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:48.024783 16330 server_base.cc:1034] running on GCE node
I20260812 02:37:48.025471 16330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:48.026216 16330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:48.027401 16330 hybrid_clock.cc:648] HybridClock initialized: now 1786502268027371 us; error 42 us; skew 500 ppm
Aug 12 02:37:48 dist-test-slave-rmhg krb5kdc[15894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502268, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:48.030768 16330 init.cc:377] Logged in from keytab as oryx/127.3.127.130@KRBTEST.COM (short username oryx)
I20260812 02:37:48.031840 16330 webserver.cc:460] Webserver started at http://127.3.127.130:37411/ using document root <none> and password file <none>
I20260812 02:37:48.032444 16330 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:48.032518 16330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:48.032755 16330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:48.035077 16330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "8e76041c93ee444c947a0b8f1d583349"
format_stamp: "Formatted at 2026-08-12 02:37:48 on dist-test-slave-rmhg"
server_key: "391ae5e1955178c32276bd41e53ef404"
server_key_iv: "f993c40fa924336699bf71d1157030a4"
server_key_version: "encryptionkey@0"
I20260812 02:37:48.035679 16330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "8e76041c93ee444c947a0b8f1d583349"
format_stamp: "Formatted at 2026-08-12 02:37:48 on dist-test-slave-rmhg"
server_key: "391ae5e1955178c32276bd41e53ef404"
server_key_iv: "f993c40fa924336699bf71d1157030a4"
server_key_version: "encryptionkey@0"
I20260812 02:37:48.039966 16330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260812 02:37:48.042938 16354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:48.044450 16330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:37:48.044608 16330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "8e76041c93ee444c947a0b8f1d583349"
format_stamp: "Formatted at 2026-08-12 02:37:48 on dist-test-slave-rmhg"
server_key: "391ae5e1955178c32276bd41e53ef404"
server_key_iv: "f993c40fa924336699bf71d1157030a4"
server_key_version: "encryptionkey@0"
I20260812 02:37:48.044739 16330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:48.065388 16330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:48.068969 16330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:48.069207 16330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:48.070964 16330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:48.071041 16330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:48.071115 16330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:48.071146 16330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:37:48 dist-test-slave-rmhg krb5kdc[15894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502268, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM
I20260812 02:37:48.082283 16330 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:45939
I20260812 02:37:48.082690 16546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:45939 every 8 connection(s)
I20260812 02:37:48.083386 16330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:37:48.086028 16047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.071826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:42175 (local address 127.3.127.190:40727)
0812 02:37:48.072048 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.072052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.073418 (+ 1366us) server_negotiation.cc:407] Connection header received
0812 02:37:48.074574 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.074578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.074649 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.074762 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:48.076318 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.077181 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.078634 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.078891 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.079011 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.081827 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:48.081831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:48.081834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:48.081873 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:48.083661 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:48.084312 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:48.084317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:48.084319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:48.084415 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:48.084780 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:48.084783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:48.084785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:48.085006 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:48.085098 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:48.085700 (+ 602us) server_negotiation.cc:299] Negotiation successful
0812 02:37:48.085847 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 02:37:48.086889 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.072618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:40727 (local address 127.3.127.130:42175)
0812 02:37:48.073251 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:48.073302 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:48.074309 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:48.074994 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:48.075004 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:48.075434 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:48.076094 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.076123 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.077331 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.077334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.078470 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.078484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.079092 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.079097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.079296 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.079959 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:48.079992 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:48.081602 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:48.083796 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:48.083804 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:48.083822 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:48.084209 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:48.084526 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:48.084529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:48.084532 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:48.084682 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:48.085112 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:48.085119 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:48.085430 (+ 311us) client_negotiation.cc:769] Sending connection context
0812 02:37:48.085633 (+ 203us) client_negotiation.cc:240] Negotiation successful
0812 02:37:48.085998 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":156,"threads_started":1}
I20260812 02:37:48.088099 16547 heartbeater.cc:347] Connected to a master server at 127.3.127.190:40727
I20260812 02:37:48.088269 16547 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:48.088833 16547 heartbeater.cc:511] Master 127.3.127.190:40727 requested a full tablet report, sending...
I20260812 02:37:48.089967 15975 ts_manager.cc:194] Registered new tserver with Master: 8e76041c93ee444c947a0b8f1d583349 (127.3.127.130:45939)
I20260812 02:37:48.090327 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 16330
I20260812 02:37:48.090444 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:37:48.090747 15975 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:42175
I20260812 02:37:48.090777 3582 external_mini_cluster.cc:1428] Setting key 1330cfcbbf7b52e9085c976bcf14de2e
I20260812 02:37:48.099853 16555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.090577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:40727 (local address 127.3.127.130:49729)
0812 02:37:48.091340 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:48.091357 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:48.091462 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:48.091689 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:48.091692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:48.091915 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:48.092059 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.092068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.093133 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.093138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.094091 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.094104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.094735 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.094739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.094895 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.095510 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:48.095530 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:48.096019 (+ 489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:48.097875 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:48.097879 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:48.097881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:48.098158 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:48.098508 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:48.098512 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:48.098514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:48.098639 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:48.099166 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:48.099170 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:48.099302 (+ 132us) client_negotiation.cc:769] Sending connection context
0812 02:37:48.099424 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 02:37:48.099616 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":130,"threads_started":1}
I20260812 02:37:48.099915 16047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.090657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:49729 (local address 127.3.127.190:40727)
0812 02:37:48.090774 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.090778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.091430 (+ 652us) server_negotiation.cc:407] Connection header received
0812 02:37:48.091547 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.091550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.091590 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.091652 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:48.092187 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.092970 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.094240 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.094582 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.094683 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.096236 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:48.096239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:48.096241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:48.096270 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:48.097740 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:48.098292 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:48.098296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:48.098297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:48.098392 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:48.098768 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:48.098772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:48.098773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:48.099053 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:48.099148 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:48.099584 (+ 436us) server_negotiation.cc:299] Negotiation successful
0812 02:37:48.099757 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for oryx/127.3.127.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.127.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.131.keytab.
Entry for principal oryx/127.3.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.131.keytab.
I20260812 02:37:48.124049 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:40727
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.131.keytab
--principal=oryx/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:36191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:48.244861 16564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:48.245221 16564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:48.245273 16564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:48.249490 16564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:48.249567 16564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:48.249611 16564 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:37:48.249644 16564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:37:48.255311 16564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:36191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.131.keytab
--principal=oryx/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.127.190:40727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:48.256477 16564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:48.257565 16564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:48.268697 16571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:48.268829 16574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:48.268904 16564 server_base.cc:1034] running on GCE node
W20260812 02:37:48.268744 16577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:48.269552 16564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:48.270291 16564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:48.271459 16564 hybrid_clock.cc:648] HybridClock initialized: now 1786502268271433 us; error 37 us; skew 500 ppm
Aug 12 02:37:48 dist-test-slave-rmhg krb5kdc[15894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502268, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:48.274459 16564 init.cc:377] Logged in from keytab as oryx/127.3.127.131@KRBTEST.COM (short username oryx)
I20260812 02:37:48.275527 16564 webserver.cc:460] Webserver started at http://127.3.127.131:44579/ using document root <none> and password file <none>
I20260812 02:37:48.276152 16564 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:48.276233 16564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:48.276463 16564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:48.278750 16564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "9e0ba3903e0c45b4894a8fc47a878e1e"
format_stamp: "Formatted at 2026-08-12 02:37:48 on dist-test-slave-rmhg"
server_key: "597dabb6514e57b117d8d9345c42bdc9"
server_key_iv: "15ef7481933442ea16a656416eb0cd3a"
server_key_version: "encryptionkey@0"
I20260812 02:37:48.279351 16564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "9e0ba3903e0c45b4894a8fc47a878e1e"
format_stamp: "Formatted at 2026-08-12 02:37:48 on dist-test-slave-rmhg"
server_key: "597dabb6514e57b117d8d9345c42bdc9"
server_key_iv: "15ef7481933442ea16a656416eb0cd3a"
server_key_version: "encryptionkey@0"
I20260812 02:37:48.283646 16564 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:37:48.286625 16585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:48.288094 16564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:37:48.288237 16564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "9e0ba3903e0c45b4894a8fc47a878e1e"
format_stamp: "Formatted at 2026-08-12 02:37:48 on dist-test-slave-rmhg"
server_key: "597dabb6514e57b117d8d9345c42bdc9"
server_key_iv: "15ef7481933442ea16a656416eb0cd3a"
server_key_version: "encryptionkey@0"
I20260812 02:37:48.288365 16564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:48.301899 16564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:48.305956 16564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:48.306174 16564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:48.307844 16564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:48.307907 16564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:48.307968 16564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:48.307997 16564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:37:48 dist-test-slave-rmhg krb5kdc[15894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502268, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM
I20260812 02:37:48.319124 16564 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:41105
I20260812 02:37:48.319545 16783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:41105 every 8 connection(s)
I20260812 02:37:48.320281 16564 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:37:48.322604 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 16564
I20260812 02:37:48.322749 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:37:48.322762 16047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.308846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:53317 (local address 127.3.127.190:40727)
0812 02:37:48.308974 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.308977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.310305 (+ 1328us) server_negotiation.cc:407] Connection header received
0812 02:37:48.311581 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.311586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.311655 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.311762 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:48.313342 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.314041 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.315386 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.315624 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.315719 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.318641 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:48.318644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:48.318646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:48.318678 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:48.320350 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:48.321002 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:48.321006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:48.321007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:48.321103 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:48.321505 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:48.321508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:48.321509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:48.321741 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:48.321819 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:48.322392 (+ 573us) server_negotiation.cc:299] Negotiation successful
0812 02:37:48.322580 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 02:37:48.323102 3582 external_mini_cluster.cc:1428] Setting key 7357819c7b647d9b3df2f31e766897e3
I20260812 02:37:48.323504 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.309540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:40727 (local address 127.3.127.131:53317)
0812 02:37:48.310144 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:48.310193 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:48.311317 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:48.312010 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:48.312021 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:48.312472 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:48.313134 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.313180 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.314173 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.314177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.315258 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.315271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.315797 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.315801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.315999 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.316608 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:48.316626 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:48.318435 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:48.320474 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:48.320480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:48.320492 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:48.320871 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:48.321234 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:48.321238 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:48.321240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:48.321421 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:48.321823 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:48.321829 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:48.322078 (+ 249us) client_negotiation.cc:769] Sending connection context
0812 02:37:48.322238 (+ 160us) client_negotiation.cc:240] Negotiation successful
0812 02:37:48.322515 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":157,"threads_started":1}
I20260812 02:37:48.324699 16785 heartbeater.cc:347] Connected to a master server at 127.3.127.190:40727
I20260812 02:37:48.324913 16785 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:48.325656 16785 heartbeater.cc:511] Master 127.3.127.190:40727 requested a full tablet report, sending...
I20260812 02:37:48.326807 15975 ts_manager.cc:194] Registered new tserver with Master: 9e0ba3903e0c45b4894a8fc47a878e1e (127.3.127.131:41105)
I20260812 02:37:48.327486 15975 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:53317
I20260812 02:37:48.334878 16047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.327286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:33889 (local address 127.3.127.190:40727)
0812 02:37:48.327416 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.327421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.327826 (+ 405us) server_negotiation.cc:407] Connection header received
0812 02:37:48.328004 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.328007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.328061 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.328125 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:48.328610 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.329177 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.330013 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.330254 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.330344 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.331554 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:48.331557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:48.331559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:48.331586 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:48.333118 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:48.333639 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:48.333643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:48.333644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:48.333731 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:48.333997 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:48.334001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:48.334002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:48.334209 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:48.334299 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:48.334570 (+ 271us) server_negotiation.cc:299] Negotiation successful
0812 02:37:48.334706 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 02:37:48.334884 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.327215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:40727 (local address 127.3.127.131:33889)
0812 02:37:48.327573 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:48.327586 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:48.327909 (+ 323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:48.328155 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:48.328157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:48.328378 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:48.328490 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.328496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.329294 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.329297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.329894 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.329904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.330366 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.330370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.330467 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.331017 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:48.331030 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:48.331391 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:48.333290 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:48.333294 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:48.333296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:48.333520 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:48.333822 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:48.333825 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:48.333827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:48.333908 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:48.334302 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:48.334305 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:48.334404 (+ 99us) client_negotiation.cc:769] Sending connection context
0812 02:37:48.334495 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 02:37:48.334670 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":95,"threads_started":1}
I20260812 02:37:48.337291 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 02:37:48 dist-test-slave-rmhg krb5kdc[15894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502268, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 02:37:48 dist-test-slave-rmhg krb5kdc[15894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:48 dist-test-slave-rmhg krb5kdc[15894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
W20260812 02:37:48.364714 16047 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:48.357876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.3.127.190:40727)
0812 02:37:48.358003 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.358007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.358021 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:37:48.358067 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.358070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.358114 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.358211 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:48.359054 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.359615 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.360534 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.360759 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.360835 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.364566 (+ 3731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49620: BlockingRecv error: recv got EOF from 127.0.0.1:49620 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 12 02:37:48 dist-test-slave-rmhg krb5kdc[15894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502268, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM
I20260812 02:37:48.375077 16047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.366449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49634 (local address 127.3.127.190:40727)
0812 02:37:48.366597 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.366600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.366795 (+ 195us) server_negotiation.cc:407] Connection header received
0812 02:37:48.366934 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.366938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.366996 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.367071 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:48.367857 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.368385 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.369281 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.369507 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.369592 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.372213 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:48.372217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:48.372219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:48.372250 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:48.373627 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:48.374032 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:48.374035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:48.374037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:48.374115 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:48.374313 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:48.374315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:48.374316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:48.374492 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:48.374629 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:48.374816 (+ 187us) server_negotiation.cc:299] Negotiation successful
0812 02:37:48.374919 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 02:37:48.378459 15975 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:37:48.382320 15975 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:37:48.397660 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.392353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.3.127.130:45939)
0812 02:37:48.392991 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.393003 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.393027 (+ 24us) server_negotiation.cc:407] Connection header received
0812 02:37:48.393105 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.393111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.393388 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.393590 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:48.393775 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.394956 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.396303 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.396888 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.397004 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.397254 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:48.397372 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 02:37:48.397484 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":116,"threads_started":1}
I20260812 02:37:48.397668 16819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.391419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:45939 (local address 127.0.0.1:43952)
0812 02:37:48.392387 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:48.392401 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:48.392490 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:48.393575 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:48.393578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:48.393597 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:48.393682 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.393687 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.395082 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.395085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.396189 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.396198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.397015 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.397019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.397139 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.397209 (+ 70us) client_negotiation.cc:769] Sending connection context
0812 02:37:48.397336 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 02:37:48.397490 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":99,"threads_started":1}
I20260812 02:37:48.398077 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.392518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47318 (local address 127.3.127.129:40483)
0812 02:37:48.392800 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.392808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.392832 (+ 24us) server_negotiation.cc:407] Connection header received
0812 02:37:48.392914 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.392919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.393210 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.393362 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:48.393684 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.394593 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.396193 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.397032 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.397623 (+ 591us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.397772 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:48.397831 (+ 59us) server_negotiation.cc:299] Negotiation successful
0812 02:37:48.397939 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":83,"threads_started":1}
I20260812 02:37:48.398124 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.391388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:40483 (local address 127.0.0.1:47318)
0812 02:37:48.391719 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:48.391781 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:48.391922 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:48.393358 (+ 1436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:48.393364 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:48.393388 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:48.393540 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.393549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.394705 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.394708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.396038 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.396049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.397214 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.397218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.397308 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.397338 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 02:37:48.397392 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 02:37:48.397487 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":3,"thread_start_us":86,"threads_started":1}
I20260812 02:37:48.398458 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.392275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35718 (local address 127.3.127.131:41105)
0812 02:37:48.393404 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.393420 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.393445 (+ 25us) server_negotiation.cc:407] Connection header received
0812 02:37:48.393522 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.393527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.393730 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.393885 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:48.394098 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.395357 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.396706 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.397506 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.397607 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.397790 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:48.398211 (+ 421us) server_negotiation.cc:299] Negotiation successful
0812 02:37:48.398316 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1018,"thread_start_us":118,"threads_started":1}
I20260812 02:37:48.401369 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.392134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:41105 (local address 127.0.0.1:35718)
0812 02:37:48.392927 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:48.392940 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:48.393024 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:48.393871 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:48.393874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:48.393894 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:48.394001 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.394008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.395479 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.395482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.396545 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.396556 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.397854 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.397857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.397965 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.397988 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 02:37:48.398057 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 02:37:48.398126 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":90,"threads_started":1}
I20260812 02:37:48.401311 16439 tablet_service.cc:1467] Processing CreateTablet for tablet 78e474fa07ed4be7ba8c2ee3dc7aeee6 (DEFAULT_TABLE table=test-table [id=c8d45c7977364a1b88b360695a32dc4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:48.401297 16207 tablet_service.cc:1467] Processing CreateTablet for tablet 78e474fa07ed4be7ba8c2ee3dc7aeee6 (DEFAULT_TABLE table=test-table [id=c8d45c7977364a1b88b360695a32dc4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:48.402575 16207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e474fa07ed4be7ba8c2ee3dc7aeee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:48.402575 16439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e474fa07ed4be7ba8c2ee3dc7aeee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:48.404316 16672 tablet_service.cc:1467] Processing CreateTablet for tablet 78e474fa07ed4be7ba8c2ee3dc7aeee6 (DEFAULT_TABLE table=test-table [id=c8d45c7977364a1b88b360695a32dc4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:48.405375 16672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e474fa07ed4be7ba8c2ee3dc7aeee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:48.410904 16832 tablet_bootstrap.cc:492] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e: Bootstrap starting.
I20260812 02:37:48.412297 16833 tablet_bootstrap.cc:492] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349: Bootstrap starting.
I20260812 02:37:48.412524 16834 tablet_bootstrap.cc:492] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e: Bootstrap starting.
I20260812 02:37:48.414207 16832 tablet_bootstrap.cc:654] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:48.414606 16833 tablet_bootstrap.cc:654] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:48.414606 16834 tablet_bootstrap.cc:654] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:48.415030 16832 log.cc:824] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:48.415468 16833 log.cc:824] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:48.415478 16834 log.cc:824] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:48.417075 16832 tablet_bootstrap.cc:492] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e: No bootstrap required, opened a new log
I20260812 02:37:48.417318 16832 ts_tablet_manager.cc:1397] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 02:37:48.417448 16834 tablet_bootstrap.cc:492] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e: No bootstrap required, opened a new log
I20260812 02:37:48.417680 16834 ts_tablet_manager.cc:1397] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 02:37:48.417909 16833 tablet_bootstrap.cc:492] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349: No bootstrap required, opened a new log
I20260812 02:37:48.418136 16833 ts_tablet_manager.cc:1397] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 02:37:48.421306 16832 raft_consensus.cc:348] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.421675 16832 raft_consensus.cc:374] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:48.421756 16832 raft_consensus.cc:729] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf33640abc844fe8199f1bd5d41a61e, State: Initialized, Role: FOLLOWER
I20260812 02:37:48.421746 16834 raft_consensus.cc:348] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.421802 16833 raft_consensus.cc:348] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.422019 16833 raft_consensus.cc:374] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:48.422078 16833 raft_consensus.cc:729] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e76041c93ee444c947a0b8f1d583349, State: Initialized, Role: FOLLOWER
I20260812 02:37:48.422132 16834 raft_consensus.cc:374] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:48.422190 16834 raft_consensus.cc:729] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e0ba3903e0c45b4894a8fc47a878e1e, State: Initialized, Role: FOLLOWER
I20260812 02:37:48.422258 16832 consensus_queue.cc:260] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.422524 16833 consensus_queue.cc:260] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.422529 16834 consensus_queue.cc:260] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.422976 16316 heartbeater.cc:503] Master 127.3.127.190:40727 was elected leader, sending a full tablet report...
I20260812 02:37:48.423383 16832 ts_tablet_manager.cc:1428] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 02:37:48.423583 16834 ts_tablet_manager.cc:1428] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 02:37:48.423583 16833 ts_tablet_manager.cc:1428] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 02:37:48.423939 16547 heartbeater.cc:503] Master 127.3.127.190:40727 was elected leader, sending a full tablet report...
I20260812 02:37:48.423956 16785 heartbeater.cc:503] Master 127.3.127.190:40727 was elected leader, sending a full tablet report...
I20260812 02:37:48.568050 16840 raft_consensus.cc:482] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:37:48.568311 16840 raft_consensus.cc:504] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.569581 16840 leader_election.cc:288] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cf33640abc844fe8199f1bd5d41a61e (127.3.127.129:40483), 8e76041c93ee444c947a0b8f1d583349 (127.3.127.130:45939)
W20260812 02:37:48.571316 16786 tablet.cc:2369] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:37:48.573993 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.569925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:40483 (local address 127.3.127.131:41481)
0812 02:37:48.570175 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:48.570191 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:48.570311 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:48.570595 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:48.570597 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:48.570619 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:48.570769 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.570777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.571826 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.571829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.572851 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.572863 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.573637 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.573641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.573733 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.573756 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 02:37:48.573817 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 02:37:48.573880 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260812 02:37:48.574054 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.570018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:41481 (local address 127.3.127.129:40483)
0812 02:37:48.570147 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.570153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.570272 (+ 119us) server_negotiation.cc:407] Connection header received
0812 02:37:48.570423 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.570426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.570486 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.570566 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:48.570923 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.571718 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.573013 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.573515 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.573645 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.573714 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:48.573847 (+ 133us) server_negotiation.cc:299] Negotiation successful
0812 02:37:48.573922 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 02:37:48.574753 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.569925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:45939 (local address 127.3.127.131:56171)
0812 02:37:48.570437 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:48.570452 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:48.570573 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:48.570906 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:48.570910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:48.570932 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:48.571516 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.571526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.572380 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.572384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.573408 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.573418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.574362 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.574365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.574457 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.574481 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 02:37:48.574557 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 02:37:48.574622 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":59,"thread_start_us":165,"threads_started":1}
I20260812 02:37:48.574798 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.570011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:56171 (local address 127.3.127.130:45939)
0812 02:37:48.570148 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.570152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.570593 (+ 441us) server_negotiation.cc:407] Connection header received
0812 02:37:48.570690 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.570693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.570745 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.570836 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:48.571666 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.572255 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.573539 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.574097 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.574231 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.574283 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:48.574587 (+ 304us) server_negotiation.cc:299] Negotiation successful
0812 02:37:48.574677 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 02:37:48.575016 16237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e474fa07ed4be7ba8c2ee3dc7aeee6" candidate_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf33640abc844fe8199f1bd5d41a61e" is_pre_election: true
I20260812 02:37:48.575383 16237 raft_consensus.cc:2454] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e0ba3903e0c45b4894a8fc47a878e1e in term 0.
I20260812 02:37:48.575573 16455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e474fa07ed4be7ba8c2ee3dc7aeee6" candidate_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e76041c93ee444c947a0b8f1d583349" is_pre_election: true
I20260812 02:37:48.575889 16455 raft_consensus.cc:2454] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e0ba3903e0c45b4894a8fc47a878e1e in term 0.
I20260812 02:37:48.576009 16594 leader_election.cc:302] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cf33640abc844fe8199f1bd5d41a61e, 9e0ba3903e0c45b4894a8fc47a878e1e; no voters:
I20260812 02:37:48.576347 16840 raft_consensus.cc:2781] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:37:48.576437 16840 raft_consensus.cc:482] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:37:48.576470 16840 raft_consensus.cc:3037] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:48.577605 16840 raft_consensus.cc:504] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.577980 16840 leader_election.cc:288] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e [CANDIDATE]: Term 1 election: Requested vote from peers 4cf33640abc844fe8199f1bd5d41a61e (127.3.127.129:40483), 8e76041c93ee444c947a0b8f1d583349 (127.3.127.130:45939)
I20260812 02:37:48.578375 16455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e474fa07ed4be7ba8c2ee3dc7aeee6" candidate_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e76041c93ee444c947a0b8f1d583349"
I20260812 02:37:48.578416 16237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e474fa07ed4be7ba8c2ee3dc7aeee6" candidate_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf33640abc844fe8199f1bd5d41a61e"
I20260812 02:37:48.578517 16455 raft_consensus.cc:3037] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:48.578540 16237 raft_consensus.cc:3037] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:48.579682 16237 raft_consensus.cc:2454] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e0ba3903e0c45b4894a8fc47a878e1e in term 1.
I20260812 02:37:48.579873 16455 raft_consensus.cc:2454] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e0ba3903e0c45b4894a8fc47a878e1e in term 1.
I20260812 02:37:48.580101 16594 leader_election.cc:302] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cf33640abc844fe8199f1bd5d41a61e, 9e0ba3903e0c45b4894a8fc47a878e1e; no voters:
I20260812 02:37:48.580312 16840 raft_consensus.cc:2781] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:48.580618 16840 raft_consensus.cc:686] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 1 LEADER]: Becoming Leader. State: Replica: 9e0ba3903e0c45b4894a8fc47a878e1e, State: Running, Role: LEADER
I20260812 02:37:48.581032 16840 consensus_queue.cc:237] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.584609 15975 catalog_manager.cc:5495] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e0ba3903e0c45b4894a8fc47a878e1e (127.3.127.131). New cstate: current_term: 1 leader_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } health_report { overall_health: HEALTHY } } }
W20260812 02:37:48.584944 16548 tablet.cc:2369] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:37:48.600402 16319 tablet.cc:2369] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:37:48.634416 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.631076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35726 (local address 127.3.127.131:41105)
0812 02:37:48.631191 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.631194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.631209 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:37:48.631315 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.631317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.631375 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.631466 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:48.631678 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.632194 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.633038 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.633303 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.633393 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.633608 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:48.634092 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:48.634185 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:48.634231 (+ 46us) server_negotiation.cc:299] Negotiation successful
0812 02:37:48.634291 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 02:37:48.640314 16455 raft_consensus.cc:1261] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349 [term 1 FOLLOWER]: Refusing update from remote peer 9e0ba3903e0c45b4894a8fc47a878e1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:48.640314 16237 raft_consensus.cc:1261] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e [term 1 FOLLOWER]: Refusing update from remote peer 9e0ba3903e0c45b4894a8fc47a878e1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:48.641114 16845 consensus_queue.cc:1035] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:48.641348 16840 consensus_queue.cc:1035] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:48.658794 15975 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:48.658977 15975 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:48.661645 15975 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea: Sending DeleteTablet for 3 replicas of tablet 78e474fa07ed4be7ba8c2ee3dc7aeee6
I20260812 02:37:48.662612 16439 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e474fa07ed4be7ba8c2ee3dc7aeee6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:48 UTC) from {username='oryx', principal='oryx/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:43952
I20260812 02:37:48.662849 16207 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e474fa07ed4be7ba8c2ee3dc7aeee6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:48 UTC) from {username='oryx', principal='oryx/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:47318
I20260812 02:37:48.663262 15975 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 02:37:48.663305 16672 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e474fa07ed4be7ba8c2ee3dc7aeee6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:48 UTC) from {username='oryx', principal='oryx/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:35718
I20260812 02:37:48.663410 16867 tablet_replica.cc:330] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e: stopping tablet replica
W20260812 02:37:48.663729 15975 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:37:48.663767 16867 raft_consensus.cc:2229] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:48.664058 16866 tablet_replica.cc:330] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349: stopping tablet replica
I20260812 02:37:48.664148 16867 raft_consensus.cc:2258] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:48.664500 16871 tablet_replica.cc:330] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e: stopping tablet replica
I20260812 02:37:48.664839 16871 raft_consensus.cc:2229] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:37:48.665179 16866 raft_consensus.cc:2229] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:48.665906 16866 raft_consensus.cc:2258] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:48.666129 16871 raft_consensus.cc:2258] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:48.668306 16867 ts_tablet_manager.cc:1905] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:48.668346 16866 ts_tablet_manager.cc:1905] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:48.669603 16207 tablet_service.cc:1467] Processing CreateTablet for tablet b6e541ff72294b8f804dddf15008f61b (DEFAULT_TABLE table=test-table [id=b28adef89e88494b8d7ea689afb77489]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:48.669967 16207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6e541ff72294b8f804dddf15008f61b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:48.671343 16867 ts_tablet_manager.cc:1918] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:37:48.671344 16866 ts_tablet_manager.cc:1918] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:37:48.671458 16866 log.cc:1196] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/78e474fa07ed4be7ba8c2ee3dc7aeee6
I20260812 02:37:48.671998 16672 tablet_service.cc:1467] Processing CreateTablet for tablet b6e541ff72294b8f804dddf15008f61b (DEFAULT_TABLE table=test-table [id=b28adef89e88494b8d7ea689afb77489]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:48.672324 16672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6e541ff72294b8f804dddf15008f61b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:48.672463 16871 ts_tablet_manager.cc:1905] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:48.674975 16834 tablet_bootstrap.cc:492] T b6e541ff72294b8f804dddf15008f61b P 9e0ba3903e0c45b4894a8fc47a878e1e: Bootstrap starting.
I20260812 02:37:48.675215 16871 ts_tablet_manager.cc:1918] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:37:48.675364 16871 log.cc:1196] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/78e474fa07ed4be7ba8c2ee3dc7aeee6
I20260812 02:37:48.675719 16871 ts_tablet_manager.cc:1939] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 9e0ba3903e0c45b4894a8fc47a878e1e: Deleting consensus metadata
I20260812 02:37:48.675704 16867 log.cc:1196] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-0/wal/wals/78e474fa07ed4be7ba8c2ee3dc7aeee6
I20260812 02:37:48.673262 16439 tablet_service.cc:1467] Processing CreateTablet for tablet b6e541ff72294b8f804dddf15008f61b (DEFAULT_TABLE table=test-table [id=b28adef89e88494b8d7ea689afb77489]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:48.676043 16439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6e541ff72294b8f804dddf15008f61b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:48.676152 16867 ts_tablet_manager.cc:1939] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 4cf33640abc844fe8199f1bd5d41a61e: Deleting consensus metadata
I20260812 02:37:48.677374 16832 tablet_bootstrap.cc:492] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e: Bootstrap starting.
I20260812 02:37:48.677654 15948 catalog_manager.cc:4841] TS 4cf33640abc844fe8199f1bd5d41a61e (127.3.127.129:40483): tablet 78e474fa07ed4be7ba8c2ee3dc7aeee6 (table test-table [id=c8d45c7977364a1b88b360695a32dc4b]) successfully deleted
I20260812 02:37:48.678342 16832 tablet_bootstrap.cc:654] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:48.678632 16833 tablet_bootstrap.cc:492] T b6e541ff72294b8f804dddf15008f61b P 8e76041c93ee444c947a0b8f1d583349: Bootstrap starting.
I20260812 02:37:48.678968 16866 ts_tablet_manager.cc:1939] T 78e474fa07ed4be7ba8c2ee3dc7aeee6 P 8e76041c93ee444c947a0b8f1d583349: Deleting consensus metadata
I20260812 02:37:48.679653 16833 tablet_bootstrap.cc:654] T b6e541ff72294b8f804dddf15008f61b P 8e76041c93ee444c947a0b8f1d583349: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:48.676151 16834 tablet_bootstrap.cc:654] T b6e541ff72294b8f804dddf15008f61b P 9e0ba3903e0c45b4894a8fc47a878e1e: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:48.680420 15945 catalog_manager.cc:4841] TS 8e76041c93ee444c947a0b8f1d583349 (127.3.127.130:45939): tablet 78e474fa07ed4be7ba8c2ee3dc7aeee6 (table test-table [id=c8d45c7977364a1b88b360695a32dc4b]) successfully deleted
I20260812 02:37:48.681473 16833 tablet_bootstrap.cc:492] T b6e541ff72294b8f804dddf15008f61b P 8e76041c93ee444c947a0b8f1d583349: No bootstrap required, opened a new log
I20260812 02:37:48.681571 16833 ts_tablet_manager.cc:1397] T b6e541ff72294b8f804dddf15008f61b P 8e76041c93ee444c947a0b8f1d583349: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:37:48.681874 16832 tablet_bootstrap.cc:492] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e: No bootstrap required, opened a new log
I20260812 02:37:48.681954 16832 ts_tablet_manager.cc:1397] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260812 02:37:48.682045 16833 raft_consensus.cc:348] T b6e541ff72294b8f804dddf15008f61b P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.682159 16833 raft_consensus.cc:374] T b6e541ff72294b8f804dddf15008f61b P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:48.682188 16833 raft_consensus.cc:729] T b6e541ff72294b8f804dddf15008f61b P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e76041c93ee444c947a0b8f1d583349, State: Initialized, Role: FOLLOWER
I20260812 02:37:48.682204 15948 catalog_manager.cc:4841] TS 9e0ba3903e0c45b4894a8fc47a878e1e (127.3.127.131:41105): tablet 78e474fa07ed4be7ba8c2ee3dc7aeee6 (table test-table [id=c8d45c7977364a1b88b360695a32dc4b]) successfully deleted
I20260812 02:37:48.682312 16833 consensus_queue.cc:260] T b6e541ff72294b8f804dddf15008f61b P 8e76041c93ee444c947a0b8f1d583349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.682396 16832 raft_consensus.cc:348] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.682508 16832 raft_consensus.cc:374] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:48.682540 16832 raft_consensus.cc:729] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf33640abc844fe8199f1bd5d41a61e, State: Initialized, Role: FOLLOWER
I20260812 02:37:48.682582 16833 ts_tablet_manager.cc:1428] T b6e541ff72294b8f804dddf15008f61b P 8e76041c93ee444c947a0b8f1d583349: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:37:48.682669 16832 consensus_queue.cc:260] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.682979 16832 ts_tablet_manager.cc:1428] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:37:48.686302 16834 tablet_bootstrap.cc:492] T b6e541ff72294b8f804dddf15008f61b P 9e0ba3903e0c45b4894a8fc47a878e1e: No bootstrap required, opened a new log
I20260812 02:37:48.686414 16834 ts_tablet_manager.cc:1397] T b6e541ff72294b8f804dddf15008f61b P 9e0ba3903e0c45b4894a8fc47a878e1e: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260812 02:37:48.686915 16834 raft_consensus.cc:348] T b6e541ff72294b8f804dddf15008f61b P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.687024 16834 raft_consensus.cc:374] T b6e541ff72294b8f804dddf15008f61b P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:48.687054 16834 raft_consensus.cc:729] T b6e541ff72294b8f804dddf15008f61b P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e0ba3903e0c45b4894a8fc47a878e1e, State: Initialized, Role: FOLLOWER
I20260812 02:37:48.687184 16834 consensus_queue.cc:260] T b6e541ff72294b8f804dddf15008f61b P 9e0ba3903e0c45b4894a8fc47a878e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.687417 16834 ts_tablet_manager.cc:1428] T b6e541ff72294b8f804dddf15008f61b P 9e0ba3903e0c45b4894a8fc47a878e1e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:37:48.704211 16839 raft_consensus.cc:482] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:37:48.704414 16839 raft_consensus.cc:504] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.705860 16839 leader_election.cc:288] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e76041c93ee444c947a0b8f1d583349 (127.3.127.130:45939), 9e0ba3903e0c45b4894a8fc47a878e1e (127.3.127.131:41105)
I20260812 02:37:48.711200 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.706537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:41287 (local address 127.3.127.131:41105)
0812 02:37:48.706690 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.706695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.706870 (+ 175us) server_negotiation.cc:407] Connection header received
0812 02:37:48.707028 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.707032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.707089 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.707174 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:48.707453 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.708249 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.709786 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.710460 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.710549 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.710606 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:48.711004 (+ 398us) server_negotiation.cc:299] Negotiation successful
0812 02:37:48.711080 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 02:37:48.711230 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.706142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:45939 (local address 127.3.127.129:54377)
0812 02:37:48.706858 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:48.706870 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:48.706941 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:48.707314 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:48.707317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:48.707331 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:48.707432 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.707439 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.708167 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.708171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.709618 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.709632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.710730 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.710734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.710851 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.710878 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 02:37:48.710954 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 02:37:48.711043 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":349,"thread_start_us":118,"threads_started":1}
I20260812 02:37:48.711287 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.706225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:54377 (local address 127.3.127.130:45939)
0812 02:37:48.706374 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.706378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.706925 (+ 547us) server_negotiation.cc:407] Connection header received
0812 02:37:48.707078 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.707082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.707130 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.707204 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:48.707523 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.708038 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.709786 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.710445 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.710549 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.710606 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:48.711044 (+ 438us) server_negotiation.cc:299] Negotiation successful
0812 02:37:48.711150 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 02:37:48.711761 16455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e541ff72294b8f804dddf15008f61b" candidate_uuid: "4cf33640abc844fe8199f1bd5d41a61e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e76041c93ee444c947a0b8f1d583349" is_pre_election: true
I20260812 02:37:48.711879 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.706138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:41105 (local address 127.3.127.129:41287)
0812 02:37:48.706768 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:48.706786 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:48.706923 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:48.707195 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:48.707198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:48.707215 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:48.707346 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.707353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.708367 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.708370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.709618 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.709632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.710741 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.710744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.710870 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.710887 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 02:37:48.711573 (+ 686us) client_negotiation.cc:240] Negotiation successful
0812 02:37:48.711668 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":32,"thread_start_us":106,"threads_started":1}
I20260812 02:37:48.711941 16455 raft_consensus.cc:2454] T b6e541ff72294b8f804dddf15008f61b P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cf33640abc844fe8199f1bd5d41a61e in term 0.
I20260812 02:37:48.712399 16706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e541ff72294b8f804dddf15008f61b" candidate_uuid: "4cf33640abc844fe8199f1bd5d41a61e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" is_pre_election: true
I20260812 02:37:48.712396 16125 leader_election.cc:302] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cf33640abc844fe8199f1bd5d41a61e, 8e76041c93ee444c947a0b8f1d583349; no voters:
I20260812 02:37:48.712652 16706 raft_consensus.cc:2454] T b6e541ff72294b8f804dddf15008f61b P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cf33640abc844fe8199f1bd5d41a61e in term 0.
I20260812 02:37:48.712921 16839 raft_consensus.cc:2781] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:37:48.712989 16839 raft_consensus.cc:482] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:37:48.713029 16839 raft_consensus.cc:3037] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:48.714130 16839 raft_consensus.cc:504] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.714512 16839 leader_election.cc:288] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e [CANDIDATE]: Term 1 election: Requested vote from peers 8e76041c93ee444c947a0b8f1d583349 (127.3.127.130:45939), 9e0ba3903e0c45b4894a8fc47a878e1e (127.3.127.131:41105)
I20260812 02:37:48.714917 16706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e541ff72294b8f804dddf15008f61b" candidate_uuid: "4cf33640abc844fe8199f1bd5d41a61e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e"
I20260812 02:37:48.714919 16455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e541ff72294b8f804dddf15008f61b" candidate_uuid: "4cf33640abc844fe8199f1bd5d41a61e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e76041c93ee444c947a0b8f1d583349"
I20260812 02:37:48.715036 16706 raft_consensus.cc:3037] T b6e541ff72294b8f804dddf15008f61b P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:48.715063 16455 raft_consensus.cc:3037] T b6e541ff72294b8f804dddf15008f61b P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:48.716017 16706 raft_consensus.cc:2454] T b6e541ff72294b8f804dddf15008f61b P 9e0ba3903e0c45b4894a8fc47a878e1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cf33640abc844fe8199f1bd5d41a61e in term 1.
I20260812 02:37:48.716228 16455 raft_consensus.cc:2454] T b6e541ff72294b8f804dddf15008f61b P 8e76041c93ee444c947a0b8f1d583349 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cf33640abc844fe8199f1bd5d41a61e in term 1.
I20260812 02:37:48.716334 16129 leader_election.cc:302] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cf33640abc844fe8199f1bd5d41a61e, 9e0ba3903e0c45b4894a8fc47a878e1e; no voters:
I20260812 02:37:48.716526 16839 raft_consensus.cc:2781] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:48.716733 16839 raft_consensus.cc:686] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e [term 1 LEADER]: Becoming Leader. State: Replica: 4cf33640abc844fe8199f1bd5d41a61e, State: Running, Role: LEADER
I20260812 02:37:48.717010 16839 consensus_queue.cc:237] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.719300 15972 catalog_manager.cc:5495] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cf33640abc844fe8199f1bd5d41a61e (127.3.127.129). New cstate: current_term: 1 leader_uuid: "4cf33640abc844fe8199f1bd5d41a61e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } health_report { overall_health: UNKNOWN } } }
Aug 12 02:37:48 dist-test-slave-rmhg krb5kdc[15894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502267, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.190@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM
I20260812 02:37:48.756225 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.747216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:40727 (local address 127.0.0.1:49648)
0812 02:37:48.747561 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:48.747577 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:48.747712 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:48.747976 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:48.747978 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:48.748436 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:48.748544 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.748551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.749584 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.749588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.750197 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.750207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.750733 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.750735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.750795 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.751351 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:48.751363 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:48.752739 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:48.754586 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:48.754596 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:48.754601 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:48.754835 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:48.755151 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:48.755153 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:48.755155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:48.755252 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:48.755706 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:48.755712 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:48.755841 (+ 129us) client_negotiation.cc:769] Sending connection context
0812 02:37:48.755941 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 02:37:48.756079 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":143,"threads_started":1}
I20260812 02:37:48.756495 16047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.747274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49648 (local address 127.3.127.190:40727)
0812 02:37:48.747398 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.747402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.747675 (+ 273us) server_negotiation.cc:407] Connection header received
0812 02:37:48.747802 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.747805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.747857 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.747943 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:48.748678 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.749478 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.750316 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.750629 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.750735 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.752892 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:48.752897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:48.752900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:48.752927 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:48.754488 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:48.754927 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:48.754930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:48.754932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:48.755023 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:48.755342 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:48.755345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:48.755346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:48.755582 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:48.755682 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:48.756018 (+ 336us) server_negotiation.cc:299] Negotiation successful
0812 02:37:48.756159 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 02:37:48.760689 15972 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:49648:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 02:37:48.761546 15972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:37:48.767110 16207 tablet_service.cc:1467] Processing CreateTablet for tablet 2280a53ded814b9caac5fe4be4658dab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c7fb07b153c54b279485401fe9fb1c6d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:37:48.767215 16439 tablet_service.cc:1467] Processing CreateTablet for tablet 2280a53ded814b9caac5fe4be4658dab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c7fb07b153c54b279485401fe9fb1c6d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:37:48.767551 16207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2280a53ded814b9caac5fe4be4658dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:48.767702 16439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2280a53ded814b9caac5fe4be4658dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:48.769788 16672 tablet_service.cc:1467] Processing CreateTablet for tablet 2280a53ded814b9caac5fe4be4658dab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c7fb07b153c54b279485401fe9fb1c6d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:37:48.770187 16672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2280a53ded814b9caac5fe4be4658dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:48.770359 16832 tablet_bootstrap.cc:492] T 2280a53ded814b9caac5fe4be4658dab P 4cf33640abc844fe8199f1bd5d41a61e: Bootstrap starting.
I20260812 02:37:48.770834 16833 tablet_bootstrap.cc:492] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349: Bootstrap starting.
I20260812 02:37:48.771348 16832 tablet_bootstrap.cc:654] T 2280a53ded814b9caac5fe4be4658dab P 4cf33640abc844fe8199f1bd5d41a61e: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:48.771848 16833 tablet_bootstrap.cc:654] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:48.772994 16834 tablet_bootstrap.cc:492] T 2280a53ded814b9caac5fe4be4658dab P 9e0ba3903e0c45b4894a8fc47a878e1e: Bootstrap starting.
I20260812 02:37:48.773523 16833 tablet_bootstrap.cc:492] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349: No bootstrap required, opened a new log
I20260812 02:37:48.773639 16833 ts_tablet_manager.cc:1397] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:37:48.774166 16834 tablet_bootstrap.cc:654] T 2280a53ded814b9caac5fe4be4658dab P 9e0ba3903e0c45b4894a8fc47a878e1e: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:48.774137 16833 raft_consensus.cc:348] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.774259 16833 raft_consensus.cc:374] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:48.774292 16833 raft_consensus.cc:729] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e76041c93ee444c947a0b8f1d583349, State: Initialized, Role: FOLLOWER
I20260812 02:37:48.774430 16833 consensus_queue.cc:260] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.774729 16833 ts_tablet_manager.cc:1428] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:37:48.775064 16841 tablet_replica.cc:439] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } } }
I20260812 02:37:48.775234 16841 tablet_replica.cc:442] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:37:48.775868 16834 tablet_bootstrap.cc:492] T 2280a53ded814b9caac5fe4be4658dab P 9e0ba3903e0c45b4894a8fc47a878e1e: No bootstrap required, opened a new log
I20260812 02:37:48.775982 16834 ts_tablet_manager.cc:1397] T 2280a53ded814b9caac5fe4be4658dab P 9e0ba3903e0c45b4894a8fc47a878e1e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:37:48.776476 16834 raft_consensus.cc:348] T 2280a53ded814b9caac5fe4be4658dab P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.776605 16834 raft_consensus.cc:374] T 2280a53ded814b9caac5fe4be4658dab P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:48.776628 16834 raft_consensus.cc:729] T 2280a53ded814b9caac5fe4be4658dab P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e0ba3903e0c45b4894a8fc47a878e1e, State: Initialized, Role: FOLLOWER
I20260812 02:37:48.777257 16832 tablet_bootstrap.cc:492] T 2280a53ded814b9caac5fe4be4658dab P 4cf33640abc844fe8199f1bd5d41a61e: No bootstrap required, opened a new log
I20260812 02:37:48.777349 16832 ts_tablet_manager.cc:1397] T 2280a53ded814b9caac5fe4be4658dab P 4cf33640abc844fe8199f1bd5d41a61e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260812 02:37:48.777742 16832 raft_consensus.cc:348] T 2280a53ded814b9caac5fe4be4658dab P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.777875 16832 raft_consensus.cc:374] T 2280a53ded814b9caac5fe4be4658dab P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:48.777920 16832 raft_consensus.cc:729] T 2280a53ded814b9caac5fe4be4658dab P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf33640abc844fe8199f1bd5d41a61e, State: Initialized, Role: FOLLOWER
I20260812 02:37:48.778059 16832 consensus_queue.cc:260] T 2280a53ded814b9caac5fe4be4658dab P 4cf33640abc844fe8199f1bd5d41a61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.778354 16832 ts_tablet_manager.cc:1428] T 2280a53ded814b9caac5fe4be4658dab P 4cf33640abc844fe8199f1bd5d41a61e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 02:37:48.778577 16834 consensus_queue.cc:260] T 2280a53ded814b9caac5fe4be4658dab P 9e0ba3903e0c45b4894a8fc47a878e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.778723 16839 tablet_replica.cc:439] T 2280a53ded814b9caac5fe4be4658dab P 4cf33640abc844fe8199f1bd5d41a61e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } } }
I20260812 02:37:48.778859 16839 tablet_replica.cc:442] T 2280a53ded814b9caac5fe4be4658dab P 4cf33640abc844fe8199f1bd5d41a61e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:37:48.778843 16840 tablet_replica.cc:439] T 2280a53ded814b9caac5fe4be4658dab P 9e0ba3903e0c45b4894a8fc47a878e1e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } } }
I20260812 02:37:48.778959 16840 tablet_replica.cc:442] T 2280a53ded814b9caac5fe4be4658dab P 9e0ba3903e0c45b4894a8fc47a878e1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:37:48.779186 16834 ts_tablet_manager.cc:1428] T 2280a53ded814b9caac5fe4be4658dab P 9e0ba3903e0c45b4894a8fc47a878e1e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260812 02:37:48.817344 16841 raft_consensus.cc:482] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:37:48.817497 16841 raft_consensus.cc:504] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.818619 16841 leader_election.cc:288] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cf33640abc844fe8199f1bd5d41a61e (127.3.127.129:40483), 9e0ba3903e0c45b4894a8fc47a878e1e (127.3.127.131:41105)
I20260812 02:37:48.823923 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.819641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:38839 (local address 127.3.127.129:40483)
0812 02:37:48.819811 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.819815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.819831 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:37:48.819892 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.819895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.819944 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.820033 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:48.820437 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.821201 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.822628 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.823248 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.823377 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.823433 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:48.823699 (+ 266us) server_negotiation.cc:299] Negotiation successful
0812 02:37:48.823793 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 02:37:48.823938 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.818849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:40483 (local address 127.3.127.130:38839)
0812 02:37:48.819156 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:48.819172 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:48.819286 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:48.820161 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:48.820164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:48.820185 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:48.820308 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.820315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.821326 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.821330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.822504 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.822516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.823378 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.823382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.823554 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.823586 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 02:37:48.823664 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 02:37:48.823743 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":58,"thread_start_us":112,"threads_started":1}
I20260812 02:37:48.823961 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.819423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:52613 (local address 127.3.127.131:41105)
0812 02:37:48.819556 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.819560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.819794 (+ 234us) server_negotiation.cc:407] Connection header received
0812 02:37:48.819915 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.819917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.819960 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.820023 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:37:48.820292 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.821076 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.822694 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.823270 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.823377 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.823433 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:48.823782 (+ 349us) server_negotiation.cc:299] Negotiation successful
0812 02:37:48.823860 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 02:37:48.824373 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.819291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:41105 (local address 127.3.127.130:52613)
0812 02:37:48.819690 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:48.819705 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:48.819817 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:48.820043 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:48.820046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:48.820064 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:37:48.820188 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.820195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.821220 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.821223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.822523 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.822591 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.823495 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.823499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.823630 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.823656 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 02:37:48.824149 (+ 493us) client_negotiation.cc:240] Negotiation successful
0812 02:37:48.824230 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":111,"thread_start_us":114,"threads_started":1}
I20260812 02:37:48.824584 16237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2280a53ded814b9caac5fe4be4658dab" candidate_uuid: "8e76041c93ee444c947a0b8f1d583349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf33640abc844fe8199f1bd5d41a61e" is_pre_election: true
I20260812 02:37:48.824757 16237 raft_consensus.cc:2454] T 2280a53ded814b9caac5fe4be4658dab P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e76041c93ee444c947a0b8f1d583349 in term 0.
I20260812 02:37:48.825191 16706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2280a53ded814b9caac5fe4be4658dab" candidate_uuid: "8e76041c93ee444c947a0b8f1d583349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" is_pre_election: true
I20260812 02:37:48.825198 16360 leader_election.cc:302] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cf33640abc844fe8199f1bd5d41a61e, 8e76041c93ee444c947a0b8f1d583349; no voters:
I20260812 02:37:48.825343 16706 raft_consensus.cc:2454] T 2280a53ded814b9caac5fe4be4658dab P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e76041c93ee444c947a0b8f1d583349 in term 0.
I20260812 02:37:48.825507 16841 raft_consensus.cc:2781] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:37:48.825574 16841 raft_consensus.cc:482] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:37:48.825615 16841 raft_consensus.cc:3037] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:48.826702 16841 raft_consensus.cc:504] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.827049 16841 leader_election.cc:288] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349 [CANDIDATE]: Term 1 election: Requested vote from peers 4cf33640abc844fe8199f1bd5d41a61e (127.3.127.129:40483), 9e0ba3903e0c45b4894a8fc47a878e1e (127.3.127.131:41105)
I20260812 02:37:48.827451 16706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2280a53ded814b9caac5fe4be4658dab" candidate_uuid: "8e76041c93ee444c947a0b8f1d583349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e"
I20260812 02:37:48.827454 16237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2280a53ded814b9caac5fe4be4658dab" candidate_uuid: "8e76041c93ee444c947a0b8f1d583349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf33640abc844fe8199f1bd5d41a61e"
I20260812 02:37:48.827581 16237 raft_consensus.cc:3037] T 2280a53ded814b9caac5fe4be4658dab P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:48.827582 16706 raft_consensus.cc:3037] T 2280a53ded814b9caac5fe4be4658dab P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:48.828728 16706 raft_consensus.cc:2454] T 2280a53ded814b9caac5fe4be4658dab P 9e0ba3903e0c45b4894a8fc47a878e1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e76041c93ee444c947a0b8f1d583349 in term 1.
I20260812 02:37:48.828728 16237 raft_consensus.cc:2454] T 2280a53ded814b9caac5fe4be4658dab P 4cf33640abc844fe8199f1bd5d41a61e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e76041c93ee444c947a0b8f1d583349 in term 1.
I20260812 02:37:48.829051 16360 leader_election.cc:302] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e76041c93ee444c947a0b8f1d583349, 9e0ba3903e0c45b4894a8fc47a878e1e; no voters:
I20260812 02:37:48.829270 16841 raft_consensus.cc:2781] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:48.829496 16841 raft_consensus.cc:686] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349 [term 1 LEADER]: Becoming Leader. State: Replica: 8e76041c93ee444c947a0b8f1d583349, State: Running, Role: LEADER
I20260812 02:37:48.829767 16841 consensus_queue.cc:237] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:48.831024 16903 tablet_replica.cc:439] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349: TxnStatusTablet state changed. Reason: New leader 8e76041c93ee444c947a0b8f1d583349. Latest consensus state: current_term: 1 leader_uuid: "8e76041c93ee444c947a0b8f1d583349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } } }
I20260812 02:37:48.831125 16903 tablet_replica.cc:442] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:37:48.831408 16909 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 02:37:48.831559 16909 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 02:37:48.832434 15973 catalog_manager.cc:5495] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e76041c93ee444c947a0b8f1d583349 (127.3.127.130). New cstate: current_term: 1 leader_uuid: "8e76041c93ee444c947a0b8f1d583349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } health_report { overall_health: UNKNOWN } } }
I20260812 02:37:48.850729 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.846127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:45939 (local address 127.0.0.1:43962)
0812 02:37:48.846342 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:48.846357 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:48.846467 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:48.846804 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:48.846807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:48.846814 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:37:48.846937 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.846943 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.847805 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.847809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.848820 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.848832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.849434 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.849439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.849546 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.849608 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:37:48.850400 (+ 792us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:37:48.850414 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 02:37:48.850480 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 02:37:48.850578 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260812 02:37:48.850721 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.846266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43962 (local address 127.3.127.130:45939)
0812 02:37:48.846390 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.846394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.846408 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:37:48.846613 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.846615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.846687 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.846772 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:48.847028 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.847677 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.848942 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.849317 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.849426 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.849760 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:48.850299 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:48.850414 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:48.850490 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 02:37:48.850553 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 02:37:48.854292 16237 raft_consensus.cc:1261] T 2280a53ded814b9caac5fe4be4658dab P 4cf33640abc844fe8199f1bd5d41a61e [term 1 FOLLOWER]: Refusing update from remote peer 8e76041c93ee444c947a0b8f1d583349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:48.854475 16706 raft_consensus.cc:1261] T 2280a53ded814b9caac5fe4be4658dab P 9e0ba3903e0c45b4894a8fc47a878e1e [term 1 FOLLOWER]: Refusing update from remote peer 8e76041c93ee444c947a0b8f1d583349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:48.854854 16841 consensus_queue.cc:1035] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:48.855334 16841 consensus_queue.cc:1035] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:48.856387 16839 tablet_replica.cc:439] T 2280a53ded814b9caac5fe4be4658dab P 4cf33640abc844fe8199f1bd5d41a61e: TxnStatusTablet state changed. Reason: New leader 8e76041c93ee444c947a0b8f1d583349. Latest consensus state: current_term: 1 leader_uuid: "8e76041c93ee444c947a0b8f1d583349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } } }
I20260812 02:37:48.856508 16839 tablet_replica.cc:442] T 2280a53ded814b9caac5fe4be4658dab P 4cf33640abc844fe8199f1bd5d41a61e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:37:48.857591 16840 tablet_replica.cc:439] T 2280a53ded814b9caac5fe4be4658dab P 9e0ba3903e0c45b4894a8fc47a878e1e: TxnStatusTablet state changed. Reason: New leader 8e76041c93ee444c947a0b8f1d583349. Latest consensus state: current_term: 1 leader_uuid: "8e76041c93ee444c947a0b8f1d583349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } } }
I20260812 02:37:48.857728 16840 tablet_replica.cc:442] T 2280a53ded814b9caac5fe4be4658dab P 9e0ba3903e0c45b4894a8fc47a878e1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:37:48.858247 16903 tablet_replica.cc:439] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8e76041c93ee444c947a0b8f1d583349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } } }
I20260812 02:37:48.858353 16903 tablet_replica.cc:442] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:37:48.860469 16839 tablet_replica.cc:439] T 2280a53ded814b9caac5fe4be4658dab P 4cf33640abc844fe8199f1bd5d41a61e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8e76041c93ee444c947a0b8f1d583349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } } }
I20260812 02:37:48.860586 16839 tablet_replica.cc:442] T 2280a53ded814b9caac5fe4be4658dab P 4cf33640abc844fe8199f1bd5d41a61e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:37:48.861313 16851 mvcc.cc:205] Tried to move back new op lower bound from 7317513293221330944 to 7317513293129416704. Current Snapshot: MvccSnapshot[applied={T|T < 7317513293221330944 or (T in {7317513293221330944})}]
I20260812 02:37:48.863260 16841 tablet_replica.cc:439] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8e76041c93ee444c947a0b8f1d583349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } } }
I20260812 02:37:48.863377 16841 tablet_replica.cc:442] T 2280a53ded814b9caac5fe4be4658dab P 8e76041c93ee444c947a0b8f1d583349: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:37:48.866317 16840 tablet_replica.cc:439] T 2280a53ded814b9caac5fe4be4658dab P 9e0ba3903e0c45b4894a8fc47a878e1e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8e76041c93ee444c947a0b8f1d583349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } } }
I20260812 02:37:48.866480 16840 tablet_replica.cc:442] T 2280a53ded814b9caac5fe4be4658dab P 9e0ba3903e0c45b4894a8fc47a878e1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:37:48.868485 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.863706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47330 (local address 127.3.127.129:40483)
0812 02:37:48.863837 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.863842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.863857 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:37:48.863930 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.863933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.863994 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.864093 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:48.864441 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.865348 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.866572 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.866885 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.866979 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.867371 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:48.867975 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:48.868086 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:48.868293 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 02:37:48.868361 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 02:37:48.879834 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.876134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:39859 (local address 127.3.127.130:45939)
0812 02:37:48.876267 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.876272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.876418 (+ 146us) server_negotiation.cc:407] Connection header received
0812 02:37:48.876556 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.876558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.876603 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.876701 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:48.876955 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.877526 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.878825 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.879034 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.879104 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.879422 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:48.879493 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:48.879539 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:48.879646 (+ 107us) server_negotiation.cc:299] Negotiation successful
0812 02:37:48.879716 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 02:37:48.879905 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.875988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:45939 (local address 127.3.127.129:39859)
0812 02:37:48.876333 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:48.876349 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:48.876465 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:48.876717 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:48.876720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:48.876728 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:37:48.876846 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.876852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.877629 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.877633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.878714 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.878727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.879135 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.879138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.879237 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.879299 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:37:48.879565 (+ 266us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:37:48.879571 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 02:37:48.879626 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 02:37:48.879716 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":117,"threads_started":1}
I20260812 02:37:48.887315 16455 raft_consensus.cc:1261] T b6e541ff72294b8f804dddf15008f61b P 8e76041c93ee444c947a0b8f1d583349 [term 1 FOLLOWER]: Refusing update from remote peer 4cf33640abc844fe8199f1bd5d41a61e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:48.887403 16706 raft_consensus.cc:1261] T b6e541ff72294b8f804dddf15008f61b P 9e0ba3903e0c45b4894a8fc47a878e1e [term 1 FOLLOWER]: Refusing update from remote peer 4cf33640abc844fe8199f1bd5d41a61e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:48.887841 16839 consensus_queue.cc:1035] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:48.888105 16881 consensus_queue.cc:1035] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:48.913214 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.909210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:36069 (local address 127.3.127.129:40483)
0812 02:37:48.909343 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.909348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.909539 (+ 191us) server_negotiation.cc:407] Connection header received
0812 02:37:48.909728 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.909731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.909792 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.909858 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:48.910249 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.911024 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.912114 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.912350 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.912438 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.912742 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:48.912826 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:48.912887 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:48.912977 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 02:37:48.913031 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 02:37:48.913235 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.909058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:40483 (local address 127.3.127.130:36069)
0812 02:37:48.909455 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:48.909471 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:48.909597 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:48.909984 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:48.909987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:48.909996 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:37:48.910122 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.910129 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.911145 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.911148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.912003 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:48.912015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.912452 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.912455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:48.912553 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.912604 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:37:48.912903 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:37:48.912910 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 02:37:48.912968 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 02:37:48.913053 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":160,"threads_started":1}
I20260812 02:37:48.937871 15973 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:48.938040 15973 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:48.940197 15973 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea: Sending DeleteTablet for 3 replicas of tablet b6e541ff72294b8f804dddf15008f61b
I20260812 02:37:48.940876 16207 tablet_service.cc:1514] Processing DeleteTablet for tablet b6e541ff72294b8f804dddf15008f61b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:48 UTC) from {username='oryx', principal='oryx/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:47318
I20260812 02:37:48.940876 16672 tablet_service.cc:1514] Processing DeleteTablet for tablet b6e541ff72294b8f804dddf15008f61b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:48 UTC) from {username='oryx', principal='oryx/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:35718
I20260812 02:37:48.941079 16871 tablet_replica.cc:330] T b6e541ff72294b8f804dddf15008f61b P 9e0ba3903e0c45b4894a8fc47a878e1e: stopping tablet replica
I20260812 02:37:48.941079 16867 tablet_replica.cc:330] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e: stopping tablet replica
I20260812 02:37:48.941224 16867 raft_consensus.cc:2229] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:37:48.941231 16871 raft_consensus.cc:2229] T b6e541ff72294b8f804dddf15008f61b P 9e0ba3903e0c45b4894a8fc47a878e1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:48.941361 16871 raft_consensus.cc:2258] T b6e541ff72294b8f804dddf15008f61b P 9e0ba3903e0c45b4894a8fc47a878e1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:48.941593 16867 raft_consensus.cc:2258] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:48.942344 16867 ts_tablet_manager.cc:1905] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:48.942418 16871 ts_tablet_manager.cc:1905] T b6e541ff72294b8f804dddf15008f61b P 9e0ba3903e0c45b4894a8fc47a878e1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:48.945540 16871 ts_tablet_manager.cc:1918] T b6e541ff72294b8f804dddf15008f61b P 9e0ba3903e0c45b4894a8fc47a878e1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:37:48.945640 16871 log.cc:1196] T b6e541ff72294b8f804dddf15008f61b P 9e0ba3903e0c45b4894a8fc47a878e1e: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/b6e541ff72294b8f804dddf15008f61b
I20260812 02:37:48.945899 16871 ts_tablet_manager.cc:1939] T b6e541ff72294b8f804dddf15008f61b P 9e0ba3903e0c45b4894a8fc47a878e1e: Deleting consensus metadata
I20260812 02:37:48.946862 15948 catalog_manager.cc:4841] TS 9e0ba3903e0c45b4894a8fc47a878e1e (127.3.127.131:41105): tablet b6e541ff72294b8f804dddf15008f61b (table test-table [id=b28adef89e88494b8d7ea689afb77489]) successfully deleted
I20260812 02:37:48.947191 16439 tablet_service.cc:1514] Processing DeleteTablet for tablet b6e541ff72294b8f804dddf15008f61b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:48 UTC) from {username='oryx', principal='oryx/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:43952
I20260812 02:37:48.947407 16866 tablet_replica.cc:330] T b6e541ff72294b8f804dddf15008f61b P 8e76041c93ee444c947a0b8f1d583349: stopping tablet replica
I20260812 02:37:48.947515 16866 raft_consensus.cc:2229] T b6e541ff72294b8f804dddf15008f61b P 8e76041c93ee444c947a0b8f1d583349 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:48.947652 16866 raft_consensus.cc:2258] T b6e541ff72294b8f804dddf15008f61b P 8e76041c93ee444c947a0b8f1d583349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:48.948766 16866 ts_tablet_manager.cc:1905] T b6e541ff72294b8f804dddf15008f61b P 8e76041c93ee444c947a0b8f1d583349: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:48.949488 16867 ts_tablet_manager.cc:1918] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:37:48.949559 16867 log.cc:1196] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-0/wal/wals/b6e541ff72294b8f804dddf15008f61b
I20260812 02:37:48.949792 16867 ts_tablet_manager.cc:1939] T b6e541ff72294b8f804dddf15008f61b P 4cf33640abc844fe8199f1bd5d41a61e: Deleting consensus metadata
I20260812 02:37:48.950572 15948 catalog_manager.cc:4841] TS 4cf33640abc844fe8199f1bd5d41a61e (127.3.127.129:40483): tablet b6e541ff72294b8f804dddf15008f61b (table test-table [id=b28adef89e88494b8d7ea689afb77489]) successfully deleted
I20260812 02:37:48.951561 16866 ts_tablet_manager.cc:1918] T b6e541ff72294b8f804dddf15008f61b P 8e76041c93ee444c947a0b8f1d583349: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:37:48.951637 16866 log.cc:1196] T b6e541ff72294b8f804dddf15008f61b P 8e76041c93ee444c947a0b8f1d583349: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/b6e541ff72294b8f804dddf15008f61b
I20260812 02:37:48.951854 16866 ts_tablet_manager.cc:1939] T b6e541ff72294b8f804dddf15008f61b P 8e76041c93ee444c947a0b8f1d583349: Deleting consensus metadata
I20260812 02:37:48.952553 15945 catalog_manager.cc:4841] TS 8e76041c93ee444c947a0b8f1d583349 (127.3.127.130:45939): tablet b6e541ff72294b8f804dddf15008f61b (table test-table [id=b28adef89e88494b8d7ea689afb77489]) successfully deleted
Aug 12 02:37:48 dist-test-slave-rmhg krb5kdc[15894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 02:37:48 dist-test-slave-rmhg krb5kdc[15894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:48 dist-test-slave-rmhg krb5kdc[15894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
W20260812 02:37:48.976402 16047 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:48.969828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49654 (local address 127.3.127.190:40727)
0812 02:37:48.969970 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.969974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.970371 (+ 397us) server_negotiation.cc:407] Connection header received
0812 02:37:48.970504 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.970507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.970557 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.970626 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:48.971377 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.971916 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.972778 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.973001 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.973092 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.976266 (+ 3174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49654: BlockingRecv error: recv got EOF from 127.0.0.1:49654 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 12 02:37:48 dist-test-slave-rmhg krb5kdc[15894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM
I20260812 02:37:48.990339 16047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:48.981620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49662 (local address 127.3.127.190:40727)
0812 02:37:48.981748 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:48.981753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:48.982036 (+ 283us) server_negotiation.cc:407] Connection header received
0812 02:37:48.982189 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:48.982192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:48.982240 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:48.982311 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:48.982968 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.983504 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.984315 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:48.984550 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:48.984645 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:48.987090 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:48.987093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:48.987095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:48.987124 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:48.988505 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:48.988946 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:48.988950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:48.988951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:48.989031 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:48.989386 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:48.989390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:48.989392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:48.989597 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:48.989693 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:48.990029 (+ 336us) server_negotiation.cc:299] Negotiation successful
0812 02:37:48.990163 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 02:37:48.993211 15973 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:37:48.993701 15973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:37:48.998768 16207 tablet_service.cc:1467] Processing CreateTablet for tablet 5977ee6ba24841ceb557a66fbeb0141e (DEFAULT_TABLE table=test-table [id=1e389f058cfb4d25924c370ae08f4bcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:48.998919 16439 tablet_service.cc:1467] Processing CreateTablet for tablet 5977ee6ba24841ceb557a66fbeb0141e (DEFAULT_TABLE table=test-table [id=1e389f058cfb4d25924c370ae08f4bcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:48.999105 16207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5977ee6ba24841ceb557a66fbeb0141e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:48.999199 16439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5977ee6ba24841ceb557a66fbeb0141e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:49.002072 16833 tablet_bootstrap.cc:492] T 5977ee6ba24841ceb557a66fbeb0141e P 8e76041c93ee444c947a0b8f1d583349: Bootstrap starting.
I20260812 02:37:49.002115 16672 tablet_service.cc:1467] Processing CreateTablet for tablet 5977ee6ba24841ceb557a66fbeb0141e (DEFAULT_TABLE table=test-table [id=1e389f058cfb4d25924c370ae08f4bcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:49.002404 16672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5977ee6ba24841ceb557a66fbeb0141e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:49.003051 16833 tablet_bootstrap.cc:654] T 5977ee6ba24841ceb557a66fbeb0141e P 8e76041c93ee444c947a0b8f1d583349: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:49.005093 16833 tablet_bootstrap.cc:492] T 5977ee6ba24841ceb557a66fbeb0141e P 8e76041c93ee444c947a0b8f1d583349: No bootstrap required, opened a new log
I20260812 02:37:49.005226 16833 ts_tablet_manager.cc:1397] T 5977ee6ba24841ceb557a66fbeb0141e P 8e76041c93ee444c947a0b8f1d583349: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:37:49.005918 16832 tablet_bootstrap.cc:492] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e: Bootstrap starting.
I20260812 02:37:49.006429 16833 raft_consensus.cc:348] T 5977ee6ba24841ceb557a66fbeb0141e P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:49.006609 16833 raft_consensus.cc:374] T 5977ee6ba24841ceb557a66fbeb0141e P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:49.006645 16833 raft_consensus.cc:729] T 5977ee6ba24841ceb557a66fbeb0141e P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e76041c93ee444c947a0b8f1d583349, State: Initialized, Role: FOLLOWER
I20260812 02:37:49.006780 16833 consensus_queue.cc:260] T 5977ee6ba24841ceb557a66fbeb0141e P 8e76041c93ee444c947a0b8f1d583349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:49.006991 16832 tablet_bootstrap.cc:654] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:49.007056 16833 ts_tablet_manager.cc:1428] T 5977ee6ba24841ceb557a66fbeb0141e P 8e76041c93ee444c947a0b8f1d583349: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 02:37:49.009327 16832 tablet_bootstrap.cc:492] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e: No bootstrap required, opened a new log
I20260812 02:37:49.009431 16832 ts_tablet_manager.cc:1397] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 02:37:49.010010 16832 raft_consensus.cc:348] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:49.010159 16832 raft_consensus.cc:374] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:49.010191 16832 raft_consensus.cc:729] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf33640abc844fe8199f1bd5d41a61e, State: Initialized, Role: FOLLOWER
I20260812 02:37:49.010222 16834 tablet_bootstrap.cc:492] T 5977ee6ba24841ceb557a66fbeb0141e P 9e0ba3903e0c45b4894a8fc47a878e1e: Bootstrap starting.
I20260812 02:37:49.010409 16832 consensus_queue.cc:260] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:49.011183 16834 tablet_bootstrap.cc:654] T 5977ee6ba24841ceb557a66fbeb0141e P 9e0ba3903e0c45b4894a8fc47a878e1e: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:49.011884 16832 ts_tablet_manager.cc:1428] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 02:37:49.014206 16834 tablet_bootstrap.cc:492] T 5977ee6ba24841ceb557a66fbeb0141e P 9e0ba3903e0c45b4894a8fc47a878e1e: No bootstrap required, opened a new log
I20260812 02:37:49.014297 16834 ts_tablet_manager.cc:1397] T 5977ee6ba24841ceb557a66fbeb0141e P 9e0ba3903e0c45b4894a8fc47a878e1e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 02:37:49.014748 16834 raft_consensus.cc:348] T 5977ee6ba24841ceb557a66fbeb0141e P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:49.014873 16834 raft_consensus.cc:374] T 5977ee6ba24841ceb557a66fbeb0141e P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:49.014910 16834 raft_consensus.cc:729] T 5977ee6ba24841ceb557a66fbeb0141e P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e0ba3903e0c45b4894a8fc47a878e1e, State: Initialized, Role: FOLLOWER
I20260812 02:37:49.015040 16834 consensus_queue.cc:260] T 5977ee6ba24841ceb557a66fbeb0141e P 9e0ba3903e0c45b4894a8fc47a878e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:49.015300 16834 ts_tablet_manager.cc:1428] T 5977ee6ba24841ceb557a66fbeb0141e P 9e0ba3903e0c45b4894a8fc47a878e1e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 02:37:49.111955 16928 raft_consensus.cc:482] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:37:49.112142 16928 raft_consensus.cc:504] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:49.112648 16928 leader_election.cc:288] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e76041c93ee444c947a0b8f1d583349 (127.3.127.130:45939), 9e0ba3903e0c45b4894a8fc47a878e1e (127.3.127.131:41105)
I20260812 02:37:49.113143 16455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5977ee6ba24841ceb557a66fbeb0141e" candidate_uuid: "4cf33640abc844fe8199f1bd5d41a61e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e76041c93ee444c947a0b8f1d583349" is_pre_election: true
I20260812 02:37:49.113372 16455 raft_consensus.cc:2454] T 5977ee6ba24841ceb557a66fbeb0141e P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cf33640abc844fe8199f1bd5d41a61e in term 0.
I20260812 02:37:49.113332 16706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5977ee6ba24841ceb557a66fbeb0141e" candidate_uuid: "4cf33640abc844fe8199f1bd5d41a61e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" is_pre_election: true
I20260812 02:37:49.113499 16706 raft_consensus.cc:2454] T 5977ee6ba24841ceb557a66fbeb0141e P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cf33640abc844fe8199f1bd5d41a61e in term 0.
I20260812 02:37:49.113752 16125 leader_election.cc:302] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cf33640abc844fe8199f1bd5d41a61e, 8e76041c93ee444c947a0b8f1d583349; no voters:
I20260812 02:37:49.113942 16928 raft_consensus.cc:2781] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:37:49.114006 16928 raft_consensus.cc:482] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:37:49.114045 16928 raft_consensus.cc:3037] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:49.115042 16928 raft_consensus.cc:504] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:49.115406 16928 leader_election.cc:288] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e [CANDIDATE]: Term 1 election: Requested vote from peers 8e76041c93ee444c947a0b8f1d583349 (127.3.127.130:45939), 9e0ba3903e0c45b4894a8fc47a878e1e (127.3.127.131:41105)
I20260812 02:37:49.115828 16455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5977ee6ba24841ceb557a66fbeb0141e" candidate_uuid: "4cf33640abc844fe8199f1bd5d41a61e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e76041c93ee444c947a0b8f1d583349"
I20260812 02:37:49.115841 16706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5977ee6ba24841ceb557a66fbeb0141e" candidate_uuid: "4cf33640abc844fe8199f1bd5d41a61e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e"
I20260812 02:37:49.115958 16455 raft_consensus.cc:3037] T 5977ee6ba24841ceb557a66fbeb0141e P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:49.115976 16706 raft_consensus.cc:3037] T 5977ee6ba24841ceb557a66fbeb0141e P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:49.116906 16455 raft_consensus.cc:2454] T 5977ee6ba24841ceb557a66fbeb0141e P 8e76041c93ee444c947a0b8f1d583349 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cf33640abc844fe8199f1bd5d41a61e in term 1.
I20260812 02:37:49.117190 16706 raft_consensus.cc:2454] T 5977ee6ba24841ceb557a66fbeb0141e P 9e0ba3903e0c45b4894a8fc47a878e1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cf33640abc844fe8199f1bd5d41a61e in term 1.
I20260812 02:37:49.117234 16125 leader_election.cc:302] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cf33640abc844fe8199f1bd5d41a61e, 8e76041c93ee444c947a0b8f1d583349; no voters:
I20260812 02:37:49.117399 16928 raft_consensus.cc:2781] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:49.117506 16928 raft_consensus.cc:686] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e [term 1 LEADER]: Becoming Leader. State: Replica: 4cf33640abc844fe8199f1bd5d41a61e, State: Running, Role: LEADER
I20260812 02:37:49.117637 16928 consensus_queue.cc:237] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:49.119624 15973 catalog_manager.cc:5495] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cf33640abc844fe8199f1bd5d41a61e (127.3.127.129). New cstate: current_term: 1 leader_uuid: "4cf33640abc844fe8199f1bd5d41a61e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } health_report { overall_health: UNKNOWN } } }
I20260812 02:37:49.140538 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:49.137727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.3.127.129:40483)
0812 02:37:49.137843 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:49.137846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:49.137877 (+ 31us) server_negotiation.cc:407] Connection header received
0812 02:37:49.137928 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:49.137930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:49.137976 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:49.138031 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:49.138223 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:49.138723 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:49.139647 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:49.139893 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:49.139986 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:49.140170 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:49.140247 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:49.140294 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:49.140362 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 02:37:49.140413 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 02:37:49.142725 16706 raft_consensus.cc:1261] T 5977ee6ba24841ceb557a66fbeb0141e P 9e0ba3903e0c45b4894a8fc47a878e1e [term 1 FOLLOWER]: Refusing update from remote peer 4cf33640abc844fe8199f1bd5d41a61e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:49.142726 16455 raft_consensus.cc:1261] T 5977ee6ba24841ceb557a66fbeb0141e P 8e76041c93ee444c947a0b8f1d583349 [term 1 FOLLOWER]: Refusing update from remote peer 4cf33640abc844fe8199f1bd5d41a61e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:49.143198 16839 consensus_queue.cc:1035] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:49.143443 16928 consensus_queue.cc:1035] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:49.147454 16850 mvcc.cc:205] Tried to move back new op lower bound from 7317513294404083712 to 7317513294307196928. Current Snapshot: MvccSnapshot[applied={T|T < 7317513294404083712}]
I20260812 02:37:49.147642 16849 mvcc.cc:205] Tried to move back new op lower bound from 7317513294404083712 to 7317513294307196928. Current Snapshot: MvccSnapshot[applied={T|T < 7317513294404083712}]
I20260812 02:37:49.151736 15973 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:49.151881 15973 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:49.153980 15973 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea: Sending DeleteTablet for 3 replicas of tablet 5977ee6ba24841ceb557a66fbeb0141e
I20260812 02:37:49.154636 16207 tablet_service.cc:1514] Processing DeleteTablet for tablet 5977ee6ba24841ceb557a66fbeb0141e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:49 UTC) from {username='oryx', principal='oryx/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:47318
I20260812 02:37:49.154644 16439 tablet_service.cc:1514] Processing DeleteTablet for tablet 5977ee6ba24841ceb557a66fbeb0141e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:49 UTC) from {username='oryx', principal='oryx/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:43952
I20260812 02:37:49.154807 16672 tablet_service.cc:1514] Processing DeleteTablet for tablet 5977ee6ba24841ceb557a66fbeb0141e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:49 UTC) from {username='oryx', principal='oryx/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:35718
I20260812 02:37:49.154891 16867 tablet_replica.cc:330] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e: stopping tablet replica
I20260812 02:37:49.154985 16871 tablet_replica.cc:330] T 5977ee6ba24841ceb557a66fbeb0141e P 9e0ba3903e0c45b4894a8fc47a878e1e: stopping tablet replica
I20260812 02:37:49.155027 16867 raft_consensus.cc:2229] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:37:49.155089 16866 tablet_replica.cc:330] T 5977ee6ba24841ceb557a66fbeb0141e P 8e76041c93ee444c947a0b8f1d583349: stopping tablet replica
I20260812 02:37:49.155119 16871 raft_consensus.cc:2229] T 5977ee6ba24841ceb557a66fbeb0141e P 9e0ba3903e0c45b4894a8fc47a878e1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:49.155220 16866 raft_consensus.cc:2229] T 5977ee6ba24841ceb557a66fbeb0141e P 8e76041c93ee444c947a0b8f1d583349 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:49.155260 16867 raft_consensus.cc:2258] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:49.155305 16871 raft_consensus.cc:2258] T 5977ee6ba24841ceb557a66fbeb0141e P 9e0ba3903e0c45b4894a8fc47a878e1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:49.155360 16866 raft_consensus.cc:2258] T 5977ee6ba24841ceb557a66fbeb0141e P 8e76041c93ee444c947a0b8f1d583349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:49.156041 16871 ts_tablet_manager.cc:1905] T 5977ee6ba24841ceb557a66fbeb0141e P 9e0ba3903e0c45b4894a8fc47a878e1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:49.156116 16866 ts_tablet_manager.cc:1905] T 5977ee6ba24841ceb557a66fbeb0141e P 8e76041c93ee444c947a0b8f1d583349: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:49.156116 16867 ts_tablet_manager.cc:1905] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:49.158185 16871 ts_tablet_manager.cc:1918] T 5977ee6ba24841ceb557a66fbeb0141e P 9e0ba3903e0c45b4894a8fc47a878e1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:37:49.158265 16871 log.cc:1196] T 5977ee6ba24841ceb557a66fbeb0141e P 9e0ba3903e0c45b4894a8fc47a878e1e: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/5977ee6ba24841ceb557a66fbeb0141e
I20260812 02:37:49.158484 16871 ts_tablet_manager.cc:1939] T 5977ee6ba24841ceb557a66fbeb0141e P 9e0ba3903e0c45b4894a8fc47a878e1e: Deleting consensus metadata
I20260812 02:37:49.158846 16867 ts_tablet_manager.cc:1918] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:37:49.158846 16866 ts_tablet_manager.cc:1918] T 5977ee6ba24841ceb557a66fbeb0141e P 8e76041c93ee444c947a0b8f1d583349: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:37:49.158932 16867 log.cc:1196] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-0/wal/wals/5977ee6ba24841ceb557a66fbeb0141e
I20260812 02:37:49.158932 16866 log.cc:1196] T 5977ee6ba24841ceb557a66fbeb0141e P 8e76041c93ee444c947a0b8f1d583349: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/5977ee6ba24841ceb557a66fbeb0141e
I20260812 02:37:49.159193 16867 ts_tablet_manager.cc:1939] T 5977ee6ba24841ceb557a66fbeb0141e P 4cf33640abc844fe8199f1bd5d41a61e: Deleting consensus metadata
I20260812 02:37:49.159192 16866 ts_tablet_manager.cc:1939] T 5977ee6ba24841ceb557a66fbeb0141e P 8e76041c93ee444c947a0b8f1d583349: Deleting consensus metadata
I20260812 02:37:49.159210 15948 catalog_manager.cc:4841] TS 9e0ba3903e0c45b4894a8fc47a878e1e (127.3.127.131:41105): tablet 5977ee6ba24841ceb557a66fbeb0141e (table test-table [id=1e389f058cfb4d25924c370ae08f4bcc]) successfully deleted
I20260812 02:37:49.159844 15948 catalog_manager.cc:4841] TS 4cf33640abc844fe8199f1bd5d41a61e (127.3.127.129:40483): tablet 5977ee6ba24841ceb557a66fbeb0141e (table test-table [id=1e389f058cfb4d25924c370ae08f4bcc]) successfully deleted
I20260812 02:37:49.159893 15945 catalog_manager.cc:4841] TS 8e76041c93ee444c947a0b8f1d583349 (127.3.127.130:45939): tablet 5977ee6ba24841ceb557a66fbeb0141e (table test-table [id=1e389f058cfb4d25924c370ae08f4bcc]) successfully deleted
I20260812 02:37:49.159950 15973 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:37:49.160385 15973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:37:49.164999 16207 tablet_service.cc:1467] Processing CreateTablet for tablet b051a527fafb4e5a85c2485b65108ba4 (DEFAULT_TABLE table=test-table [id=52b62b8e0a314a5ba8dc814f5396d4fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:49.165359 16207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b051a527fafb4e5a85c2485b65108ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:49.165349 16439 tablet_service.cc:1467] Processing CreateTablet for tablet b051a527fafb4e5a85c2485b65108ba4 (DEFAULT_TABLE table=test-table [id=52b62b8e0a314a5ba8dc814f5396d4fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:49.165349 16672 tablet_service.cc:1467] Processing CreateTablet for tablet b051a527fafb4e5a85c2485b65108ba4 (DEFAULT_TABLE table=test-table [id=52b62b8e0a314a5ba8dc814f5396d4fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:37:49.165650 16439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b051a527fafb4e5a85c2485b65108ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:49.165658 16672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b051a527fafb4e5a85c2485b65108ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:49.167546 16832 tablet_bootstrap.cc:492] T b051a527fafb4e5a85c2485b65108ba4 P 4cf33640abc844fe8199f1bd5d41a61e: Bootstrap starting.
I20260812 02:37:49.168044 16833 tablet_bootstrap.cc:492] T b051a527fafb4e5a85c2485b65108ba4 P 8e76041c93ee444c947a0b8f1d583349: Bootstrap starting.
I20260812 02:37:49.168390 16834 tablet_bootstrap.cc:492] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e: Bootstrap starting.
I20260812 02:37:49.168403 16832 tablet_bootstrap.cc:654] T b051a527fafb4e5a85c2485b65108ba4 P 4cf33640abc844fe8199f1bd5d41a61e: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:49.168875 16833 tablet_bootstrap.cc:654] T b051a527fafb4e5a85c2485b65108ba4 P 8e76041c93ee444c947a0b8f1d583349: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:49.169245 16834 tablet_bootstrap.cc:654] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:49.170018 16832 tablet_bootstrap.cc:492] T b051a527fafb4e5a85c2485b65108ba4 P 4cf33640abc844fe8199f1bd5d41a61e: No bootstrap required, opened a new log
I20260812 02:37:49.170065 16833 tablet_bootstrap.cc:492] T b051a527fafb4e5a85c2485b65108ba4 P 8e76041c93ee444c947a0b8f1d583349: No bootstrap required, opened a new log
I20260812 02:37:49.170110 16832 ts_tablet_manager.cc:1397] T b051a527fafb4e5a85c2485b65108ba4 P 4cf33640abc844fe8199f1bd5d41a61e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 02:37:49.170172 16833 ts_tablet_manager.cc:1397] T b051a527fafb4e5a85c2485b65108ba4 P 8e76041c93ee444c947a0b8f1d583349: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 02:37:49.170403 16834 tablet_bootstrap.cc:492] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e: No bootstrap required, opened a new log
I20260812 02:37:49.170497 16834 ts_tablet_manager.cc:1397] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 02:37:49.170531 16832 raft_consensus.cc:348] T b051a527fafb4e5a85c2485b65108ba4 P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:49.170665 16832 raft_consensus.cc:374] T b051a527fafb4e5a85c2485b65108ba4 P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:49.170626 16833 raft_consensus.cc:348] T b051a527fafb4e5a85c2485b65108ba4 P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:49.170708 16832 raft_consensus.cc:729] T b051a527fafb4e5a85c2485b65108ba4 P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf33640abc844fe8199f1bd5d41a61e, State: Initialized, Role: FOLLOWER
I20260812 02:37:49.170742 16833 raft_consensus.cc:374] T b051a527fafb4e5a85c2485b65108ba4 P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:49.170780 16833 raft_consensus.cc:729] T b051a527fafb4e5a85c2485b65108ba4 P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e76041c93ee444c947a0b8f1d583349, State: Initialized, Role: FOLLOWER
I20260812 02:37:49.170835 16832 consensus_queue.cc:260] T b051a527fafb4e5a85c2485b65108ba4 P 4cf33640abc844fe8199f1bd5d41a61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:49.170913 16833 consensus_queue.cc:260] T b051a527fafb4e5a85c2485b65108ba4 P 8e76041c93ee444c947a0b8f1d583349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:49.170925 16834 raft_consensus.cc:348] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:49.171018 16834 raft_consensus.cc:374] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:49.171053 16834 raft_consensus.cc:729] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e0ba3903e0c45b4894a8fc47a878e1e, State: Initialized, Role: FOLLOWER
I20260812 02:37:49.171109 16832 ts_tablet_manager.cc:1428] T b051a527fafb4e5a85c2485b65108ba4 P 4cf33640abc844fe8199f1bd5d41a61e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 02:37:49.171203 16833 ts_tablet_manager.cc:1428] T b051a527fafb4e5a85c2485b65108ba4 P 8e76041c93ee444c947a0b8f1d583349: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:37:49.171182 16834 consensus_queue.cc:260] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:49.171895 16834 ts_tablet_manager.cc:1428] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 02:37:49.419077 16840 raft_consensus.cc:482] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:37:49.419238 16840 raft_consensus.cc:504] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:49.419708 16840 leader_election.cc:288] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cf33640abc844fe8199f1bd5d41a61e (127.3.127.129:40483), 8e76041c93ee444c947a0b8f1d583349 (127.3.127.130:45939)
I20260812 02:37:49.420197 16237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b051a527fafb4e5a85c2485b65108ba4" candidate_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf33640abc844fe8199f1bd5d41a61e" is_pre_election: true
I20260812 02:37:49.420269 16455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b051a527fafb4e5a85c2485b65108ba4" candidate_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e76041c93ee444c947a0b8f1d583349" is_pre_election: true
I20260812 02:37:49.420354 16237 raft_consensus.cc:2454] T b051a527fafb4e5a85c2485b65108ba4 P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e0ba3903e0c45b4894a8fc47a878e1e in term 0.
I20260812 02:37:49.420455 16455 raft_consensus.cc:2454] T b051a527fafb4e5a85c2485b65108ba4 P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e0ba3903e0c45b4894a8fc47a878e1e in term 0.
I20260812 02:37:49.420675 16594 leader_election.cc:302] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cf33640abc844fe8199f1bd5d41a61e, 9e0ba3903e0c45b4894a8fc47a878e1e; no voters:
I20260812 02:37:49.420852 16840 raft_consensus.cc:2781] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:37:49.420918 16840 raft_consensus.cc:482] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:37:49.420953 16840 raft_consensus.cc:3037] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:49.421980 16840 raft_consensus.cc:504] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:49.422350 16840 leader_election.cc:288] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e [CANDIDATE]: Term 1 election: Requested vote from peers 4cf33640abc844fe8199f1bd5d41a61e (127.3.127.129:40483), 8e76041c93ee444c947a0b8f1d583349 (127.3.127.130:45939)
I20260812 02:37:49.422778 16237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b051a527fafb4e5a85c2485b65108ba4" candidate_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf33640abc844fe8199f1bd5d41a61e"
I20260812 02:37:49.422802 16455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b051a527fafb4e5a85c2485b65108ba4" candidate_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e76041c93ee444c947a0b8f1d583349"
I20260812 02:37:49.422904 16455 raft_consensus.cc:3037] T b051a527fafb4e5a85c2485b65108ba4 P 8e76041c93ee444c947a0b8f1d583349 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:49.422915 16237 raft_consensus.cc:3037] T b051a527fafb4e5a85c2485b65108ba4 P 4cf33640abc844fe8199f1bd5d41a61e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:49.423794 16455 raft_consensus.cc:2454] T b051a527fafb4e5a85c2485b65108ba4 P 8e76041c93ee444c947a0b8f1d583349 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e0ba3903e0c45b4894a8fc47a878e1e in term 1.
I20260812 02:37:49.423867 16237 raft_consensus.cc:2454] T b051a527fafb4e5a85c2485b65108ba4 P 4cf33640abc844fe8199f1bd5d41a61e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e0ba3903e0c45b4894a8fc47a878e1e in term 1.
I20260812 02:37:49.424093 16590 leader_election.cc:302] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e76041c93ee444c947a0b8f1d583349, 9e0ba3903e0c45b4894a8fc47a878e1e; no voters:
I20260812 02:37:49.424288 16840 raft_consensus.cc:2781] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:49.424449 16840 raft_consensus.cc:686] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 1 LEADER]: Becoming Leader. State: Replica: 9e0ba3903e0c45b4894a8fc47a878e1e, State: Running, Role: LEADER
I20260812 02:37:49.424620 16840 consensus_queue.cc:237] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } }
I20260812 02:37:49.426318 15975 catalog_manager.cc:5495] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e0ba3903e0c45b4894a8fc47a878e1e (127.3.127.131). New cstate: current_term: 1 leader_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e0ba3903e0c45b4894a8fc47a878e1e" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41105 } health_report { overall_health: HEALTHY } } }
I20260812 02:37:49.474900 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:49.471696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35734 (local address 127.3.127.131:41105)
0812 02:37:49.471965 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:49.471970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:49.471982 (+ 12us) server_negotiation.cc:407] Connection header received
0812 02:37:49.472024 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:49.472026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:49.472068 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:49.472141 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:49.472403 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:49.472949 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:49.473809 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:49.474046 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:49.474136 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:49.474394 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:49.474499 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:49.474566 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:49.474675 (+ 109us) server_negotiation.cc:299] Negotiation successful
0812 02:37:49.474737 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":96,"threads_started":1}
I20260812 02:37:49.485021 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:49.481841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:45939 (local address 127.3.127.131:58813)
0812 02:37:49.482164 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:49.482179 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:49.482268 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:49.482491 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:49.482493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:49.482499 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:37:49.482589 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:49.482593 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:49.483291 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:49.483294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:49.483916 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:49.483924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:49.484353 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:49.484357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:49.484428 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:49.484489 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:37:49.484765 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:37:49.484770 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 02:37:49.484815 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 02:37:49.484894 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":132,"threads_started":1}
I20260812 02:37:49.485024 16979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:49.482000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:58813 (local address 127.3.127.130:45939)
0812 02:37:49.482257 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:49.482261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:49.482276 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:37:49.482367 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:49.482369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:49.482414 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:49.482478 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:49.482689 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:49.483216 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:49.484005 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:49.484252 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:49.484326 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:49.484602 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:49.484701 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:49.484751 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:49.484833 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 02:37:49.484888 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
I20260812 02:37:49.491593 16455 raft_consensus.cc:1261] T b051a527fafb4e5a85c2485b65108ba4 P 8e76041c93ee444c947a0b8f1d583349 [term 1 FOLLOWER]: Refusing update from remote peer 9e0ba3903e0c45b4894a8fc47a878e1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:49.491626 16237 raft_consensus.cc:1261] T b051a527fafb4e5a85c2485b65108ba4 P 4cf33640abc844fe8199f1bd5d41a61e [term 1 FOLLOWER]: Refusing update from remote peer 9e0ba3903e0c45b4894a8fc47a878e1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:37:49.491974 16970 consensus_queue.cc:1035] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cf33640abc844fe8199f1bd5d41a61e" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:49.492125 16840 consensus_queue.cc:1035] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e76041c93ee444c947a0b8f1d583349" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 45939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:37:49.510191 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:49.506885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:36703 (local address 127.3.127.131:41105)
0812 02:37:49.507024 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:49.507028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:49.507096 (+ 68us) server_negotiation.cc:407] Connection header received
0812 02:37:49.507239 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:49.507242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:49.507295 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:49.507356 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:37:49.507554 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:49.508077 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:49.509027 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:49.509357 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:49.509455 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:49.509733 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:37:49.509806 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:37:49.509865 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:49.509974 (+ 109us) server_negotiation.cc:299] Negotiation successful
0812 02:37:49.510033 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 02:37:49.510205 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:49.506731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:41105 (local address 127.3.127.130:36703)
0812 02:37:49.507025 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:49.507040 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:49.507143 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:49.507374 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:49.507376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:49.507383 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:37:49.507480 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:49.507484 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:49.508166 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:49.508170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:49.508909 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:49.508922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:49.509466 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:49.509470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:49.509555 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:49.509588 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:37:49.509886 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:37:49.509895 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 02:37:49.509953 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 02:37:49.510045 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":119,"threads_started":1}
I20260812 02:37:49.529805 15975 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:49.529955 15975 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:37:49.531844 15975 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe2c7796e32049f2ba7710104f5520ea: Sending DeleteTablet for 3 replicas of tablet b051a527fafb4e5a85c2485b65108ba4
I20260812 02:37:49.532454 16207 tablet_service.cc:1514] Processing DeleteTablet for tablet b051a527fafb4e5a85c2485b65108ba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:49 UTC) from {username='oryx', principal='oryx/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:47318
I20260812 02:37:49.532542 16439 tablet_service.cc:1514] Processing DeleteTablet for tablet b051a527fafb4e5a85c2485b65108ba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:49 UTC) from {username='oryx', principal='oryx/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:43952
I20260812 02:37:49.532666 16672 tablet_service.cc:1514] Processing DeleteTablet for tablet b051a527fafb4e5a85c2485b65108ba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:37:49 UTC) from {username='oryx', principal='oryx/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:35718
I20260812 02:37:49.532753 16866 tablet_replica.cc:330] T b051a527fafb4e5a85c2485b65108ba4 P 8e76041c93ee444c947a0b8f1d583349: stopping tablet replica
I20260812 02:37:49.532845 16867 tablet_replica.cc:330] T b051a527fafb4e5a85c2485b65108ba4 P 4cf33640abc844fe8199f1bd5d41a61e: stopping tablet replica
I20260812 02:37:49.532876 16866 raft_consensus.cc:2229] T b051a527fafb4e5a85c2485b65108ba4 P 8e76041c93ee444c947a0b8f1d583349 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:49.532967 16867 raft_consensus.cc:2229] T b051a527fafb4e5a85c2485b65108ba4 P 4cf33640abc844fe8199f1bd5d41a61e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:37:49.532992 16866 raft_consensus.cc:2258] T b051a527fafb4e5a85c2485b65108ba4 P 8e76041c93ee444c947a0b8f1d583349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:49.533090 16867 raft_consensus.cc:2258] T b051a527fafb4e5a85c2485b65108ba4 P 4cf33640abc844fe8199f1bd5d41a61e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:49.533272 16871 tablet_replica.cc:330] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e: stopping tablet replica
I20260812 02:37:49.533373 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 16090
I20260812 02:37:49.533390 16871 raft_consensus.cc:2229] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:37:49.533567 16871 raft_consensus.cc:2258] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:37:49.537112 16871 ts_tablet_manager.cc:1905] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:49.537262 16866 ts_tablet_manager.cc:1905] T b051a527fafb4e5a85c2485b65108ba4 P 8e76041c93ee444c947a0b8f1d583349: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:37:49.539737 16871 ts_tablet_manager.cc:1918] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:37:49.539737 16866 ts_tablet_manager.cc:1918] T b051a527fafb4e5a85c2485b65108ba4 P 8e76041c93ee444c947a0b8f1d583349: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:37:49.539809 16866 log.cc:1196] T b051a527fafb4e5a85c2485b65108ba4 P 8e76041c93ee444c947a0b8f1d583349: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/b051a527fafb4e5a85c2485b65108ba4
I20260812 02:37:49.539809 16871 log.cc:1196] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/b051a527fafb4e5a85c2485b65108ba4
I20260812 02:37:49.540055 16866 ts_tablet_manager.cc:1939] T b051a527fafb4e5a85c2485b65108ba4 P 8e76041c93ee444c947a0b8f1d583349: Deleting consensus metadata
I20260812 02:37:49.540055 16871 ts_tablet_manager.cc:1939] T b051a527fafb4e5a85c2485b65108ba4 P 9e0ba3903e0c45b4894a8fc47a878e1e: Deleting consensus metadata
I20260812 02:37:49.540877 15945 catalog_manager.cc:4841] TS 8e76041c93ee444c947a0b8f1d583349 (127.3.127.130:45939): tablet b051a527fafb4e5a85c2485b65108ba4 (table test-table [id=52b62b8e0a314a5ba8dc814f5396d4fb]) successfully deleted
I20260812 02:37:49.543833 15948 catalog_manager.cc:4841] TS 9e0ba3903e0c45b4894a8fc47a878e1e (127.3.127.131:41105): tablet b051a527fafb4e5a85c2485b65108ba4 (table test-table [id=52b62b8e0a314a5ba8dc814f5396d4fb]) successfully deleted
W20260812 02:37:49.553522 15948 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.127.129:40483 (error 108)
I20260812 02:37:49.553846 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 16330
I20260812 02:37:49.565567 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:49.565112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:40483 (local address 127.0.0.1:47340)
0812 02:37:49.565392 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:49.565475 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.129:40483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":115,"threads_started":1}
W20260812 02:37:49.565757 15948 catalog_manager.cc:4584] TS 4cf33640abc844fe8199f1bd5d41a61e (127.3.127.129:40483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b051a527fafb4e5a85c2485b65108ba4: Network error: Client connection negotiation failed: client connection to 127.3.127.129:40483: connect: Connection refused (error 111)
I20260812 02:37:49.571882 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 16564
I20260812 02:37:49.587427 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 15920
2026-08-12T02:37:49Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3578 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:37:49 dist-test-slave-rmhg krb5kdc[16997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:37:49 dist-test-slave-rmhg krb5kdc[16997](info): set up 2 sockets
Aug 12 02:37:49 dist-test-slave-rmhg krb5kdc[16997](info): commencing operation
krb5kdc: starting...
W20260812 02:37:50.778040 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:37:50 dist-test-slave-rmhg krb5kdc[16997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:37:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:37:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:37:50.910209 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:44335
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:40867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:44335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:51.028654 17020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:51.028985 17020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:51.029050 17020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:51.033458 17020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:51.033564 17020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:51.033598 17020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:51.033622 17020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:51.033643 17020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:37:51.039395 17020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:40867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:44335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:44335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:51.040597 17020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:51.041724 17020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:37:51.051010 17020 server_base.cc:1034] running on GCE node
W20260812 02:37:51.050937 17034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:51.050930 17028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:51.050927 17030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:51.051999 17020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:51.053236 17020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:51.054411 17020 hybrid_clock.cc:648] HybridClock initialized: now 1786502271054397 us; error 26 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T02:37:51Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1469 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:37:51 dist-test-slave-rmhg krb5kdc[17043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:37:51 dist-test-slave-rmhg krb5kdc[17043](info): set up 2 sockets
Aug 12 02:37:51 dist-test-slave-rmhg krb5kdc[17043](info): commencing operation
krb5kdc: starting...
W20260812 02:37:52.243048 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:37:52 dist-test-slave-rmhg krb5kdc[17043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:37:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:37:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:37:52.379223 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:41875
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:33817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:41875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:52.498804 17071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:52.499147 17071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:52.499193 17071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:52.503423 17071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:52.503522 17071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:52.503542 17071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:52.503558 17071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:52.503573 17071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:37:52.509485 17071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:33817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:41875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:41875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:52.510665 17071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:52.511725 17071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:52.521095 17081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:52.521181 17083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:52.521301 17071 server_base.cc:1034] running on GCE node
W20260812 02:37:52.521555 17080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:52.522162 17071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:52.523398 17071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:52.524582 17071 hybrid_clock.cc:648] HybridClock initialized: now 1786502272524562 us; error 38 us; skew 500 ppm
Aug 12 02:37:52 dist-test-slave-rmhg krb5kdc[17043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502272, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:52.527595 17071 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:37:52.528644 17071 webserver.cc:460] Webserver started at http://127.3.127.190:45923/ using document root <none> and password file <none>
I20260812 02:37:52.529285 17071 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:52.529367 17071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:52.529594 17071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:52.531786 17071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "e47e5630f715400b8b6f537394e88905"
format_stamp: "Formatted at 2026-08-12 02:37:52 on dist-test-slave-rmhg"
server_key: "5c9336c047e39ee0189370f71c99fa7e"
server_key_iv: "11b4dfc586c44188c5fb11bc27835959"
server_key_version: "encryptionkey@0"
I20260812 02:37:52.532428 17071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "e47e5630f715400b8b6f537394e88905"
format_stamp: "Formatted at 2026-08-12 02:37:52 on dist-test-slave-rmhg"
server_key: "5c9336c047e39ee0189370f71c99fa7e"
server_key_iv: "11b4dfc586c44188c5fb11bc27835959"
server_key_version: "encryptionkey@0"
I20260812 02:37:52.536574 17071 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:37:52.539263 17097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:52.540542 17071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 02:37:52.540673 17071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "e47e5630f715400b8b6f537394e88905"
format_stamp: "Formatted at 2026-08-12 02:37:52 on dist-test-slave-rmhg"
server_key: "5c9336c047e39ee0189370f71c99fa7e"
server_key_iv: "11b4dfc586c44188c5fb11bc27835959"
server_key_version: "encryptionkey@0"
I20260812 02:37:52.540786 17071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:52.566210 17071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:52.569469 17071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:52.569685 17071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:52.577304 17071 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:41875
I20260812 02:37:52.577318 17179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:41875 every 8 connection(s)
I20260812 02:37:52.578470 17071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:37:52.581773 17182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:37:52.587962 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 17071
I20260812 02:37:52.588115 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:37:52.588414 3582 external_mini_cluster.cc:1428] Setting key 76b91cea6dc9b4ca32b95add36b3d054
I20260812 02:37:52.588477 17182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905: Bootstrap starting.
I20260812 02:37:52.591382 17182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905: Neither blocks nor log segments found. Creating new log.
I20260812 02:37:52.592391 17182 log.cc:824] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:52.594964 17182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905: No bootstrap required, opened a new log
Aug 12 02:37:52 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:52.599831 17182 raft_consensus.cc:348] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47e5630f715400b8b6f537394e88905" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41875 } }
I20260812 02:37:52.600133 17182 raft_consensus.cc:374] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:37:52.600207 17182 raft_consensus.cc:729] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47e5630f715400b8b6f537394e88905, State: Initialized, Role: FOLLOWER
I20260812 02:37:52.600710 17182 consensus_queue.cc:260] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47e5630f715400b8b6f537394e88905" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41875 } }
I20260812 02:37:52.600855 17182 raft_consensus.cc:388] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:37:52.600929 17182 raft_consensus.cc:482] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:37:52.601040 17182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:37:52.602312 17182 raft_consensus.cc:504] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47e5630f715400b8b6f537394e88905" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41875 } }
I20260812 02:37:52.602722 17182 leader_election.cc:302] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e47e5630f715400b8b6f537394e88905; no voters:
I20260812 02:37:52.603113 17182 leader_election.cc:288] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:37:52.603246 17193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:37:52.603494 17193 raft_consensus.cc:686] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [term 1 LEADER]: Becoming Leader. State: Replica: e47e5630f715400b8b6f537394e88905, State: Running, Role: LEADER
I20260812 02:37:52.603889 17193 consensus_queue.cc:237] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47e5630f715400b8b6f537394e88905" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41875 } }
I20260812 02:37:52.604709 17182 sys_catalog.cc:564] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:37:52.605334 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:52.589944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57592 (local address 127.3.127.190:41875)
0812 02:37:52.590409 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:52.590426 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:52.590466 (+ 40us) server_negotiation.cc:407] Connection header received
0812 02:37:52.591203 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:52.591225 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:52.591635 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:52.591967 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:52.592932 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:52.594169 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:52.595372 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:52.595895 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:52.596037 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:52.599206 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:52.599214 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:52.599233 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:52.599276 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:52.601265 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:52.601850 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:52.601857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:52.601863 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:52.602010 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:52.602344 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:52.602346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:52.602347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:52.602859 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:52.603030 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:52.603873 (+ 843us) server_negotiation.cc:299] Negotiation successful
0812 02:37:52.604225 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":135,"threads_started":1}
I20260812 02:37:52.605583 17195 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e47e5630f715400b8b6f537394e88905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47e5630f715400b8b6f537394e88905" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41875 } } }
I20260812 02:37:52.605736 17195 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:52.606079 17196 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e47e5630f715400b8b6f537394e88905. Latest consensus state: current_term: 1 leader_uuid: "e47e5630f715400b8b6f537394e88905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47e5630f715400b8b6f537394e88905" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41875 } } }
I20260812 02:37:52.606146 17211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:37:52.606166 17196 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:52.609678 17211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:37:52.615617 17211 catalog_manager.cc:1347] Generated new cluster ID: 189dcfe291e64388b14416b27cc9afcc
I20260812 02:37:52.615694 17211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:37:52.631475 17211 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:37:52.632578 17211 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:37:52.647408 17211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905: Generated new TSK 0
I20260812 02:37:52.648262 17211 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
I20260812 02:37:52.684046 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:41875
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:33817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:52.819139 17234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:52.819460 17234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:52.819507 17234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:52.823730 17234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:52.823809 17234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:52.823876 17234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:37:52.829563 17234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:33817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:41875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:52.830829 17234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:52.831974 17234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:52.842435 17250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:52.842437 17249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:52.842470 17254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:52.842808 17234 server_base.cc:1034] running on GCE node
I20260812 02:37:52.843251 17234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:52.843981 17234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:52.845193 17234 hybrid_clock.cc:648] HybridClock initialized: now 1786502272845132 us; error 41 us; skew 500 ppm
Aug 12 02:37:52 dist-test-slave-rmhg krb5kdc[17043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502272, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:52.847994 17234 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:37:52.848956 17234 webserver.cc:460] Webserver started at http://127.3.127.129:43295/ using document root <none> and password file <none>
I20260812 02:37:52.849574 17234 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:52.849649 17234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:52.849869 17234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:52.852034 17234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "1565e6caf697459dbeac744731b13bd5"
format_stamp: "Formatted at 2026-08-12 02:37:52 on dist-test-slave-rmhg"
server_key: "48f20196f0c1c2acddbf6173f2e01825"
server_key_iv: "3361a11e578add1ec4839ce04bdaa249"
server_key_version: "encryptionkey@0"
I20260812 02:37:52.852633 17234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "1565e6caf697459dbeac744731b13bd5"
format_stamp: "Formatted at 2026-08-12 02:37:52 on dist-test-slave-rmhg"
server_key: "48f20196f0c1c2acddbf6173f2e01825"
server_key_iv: "3361a11e578add1ec4839ce04bdaa249"
server_key_version: "encryptionkey@0"
I20260812 02:37:52.856629 17234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:37:52.859314 17268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:52.860642 17234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 02:37:52.860762 17234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "1565e6caf697459dbeac744731b13bd5"
format_stamp: "Formatted at 2026-08-12 02:37:52 on dist-test-slave-rmhg"
server_key: "48f20196f0c1c2acddbf6173f2e01825"
server_key_iv: "3361a11e578add1ec4839ce04bdaa249"
server_key_version: "encryptionkey@0"
I20260812 02:37:52.860864 17234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:52.870066 17234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:52.873191 17234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:52.873377 17234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:52.873994 17234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:52.874856 17234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:52.874904 17234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:52.874960 17234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:52.874986 17234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:52.883975 17234 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:43613
I20260812 02:37:52.884011 17465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:43613 every 8 connection(s)
I20260812 02:37:52.885021 17234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:37:52.892436 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 17234
I20260812 02:37:52.892576 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:37:52.892902 3582 external_mini_cluster.cc:1428] Setting key 62d82bbcdaebe886f7954b59d8ca320f
Aug 12 02:37:52 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502272, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:52.900161 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:52.887112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:47995 (local address 127.3.127.190:41875)
0812 02:37:52.887254 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:52.887258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:52.888155 (+ 897us) server_negotiation.cc:407] Connection header received
0812 02:37:52.889584 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:52.889587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:52.889653 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:52.889730 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:52.891019 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:52.891574 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:52.893137 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:52.893482 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:52.893597 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:52.896596 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:52.896600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:52.896602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:52.896631 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:52.898035 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:52.898582 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:52.898586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:52.898588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:52.898677 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:52.899013 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:52.899017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:52.899018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:52.899219 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:52.899308 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:52.899853 (+ 545us) server_negotiation.cc:299] Negotiation successful
0812 02:37:52.899989 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 02:37:52.901054 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:52.887509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.129:47995)
0812 02:37:52.888003 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:52.888051 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:52.889357 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:52.889933 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:52.889942 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:52.890327 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:52.890863 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:52.890878 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:52.891716 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:52.891719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:52.892970 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:52.892984 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:52.893678 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:52.893683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:52.893887 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:52.894553 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:52.894576 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:52.896394 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:52.898169 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:52.898176 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:52.898195 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:52.898479 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:52.898764 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:52.898768 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:52.898770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:52.898921 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:52.899324 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:52.899331 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:52.899607 (+ 276us) client_negotiation.cc:769] Sending connection context
0812 02:37:52.899795 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 02:37:52.900171 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":126,"threads_started":1}
I20260812 02:37:52.902510 17466 heartbeater.cc:347] Connected to a master server at 127.3.127.190:41875
I20260812 02:37:52.902776 17466 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:52.903352 17466 heartbeater.cc:511] Master 127.3.127.190:41875 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.127.130@KRBTEST.COM; defaulting to no policy
I20260812 02:37:52.905083 17121 ts_manager.cc:194] Registered new tserver with Master: 1565e6caf697459dbeac744731b13bd5 (127.3.127.129:43613)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.130@KRBTEST.COM" created.
I20260812 02:37:52.906665 17121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:47995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
I20260812 02:37:52.922250 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:41875
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:33817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:53.042158 17478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:53.042438 17478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:53.042486 17478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:53.046700 17478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:53.046772 17478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:53.046844 17478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:37:53.052412 17478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:33817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:41875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:53.053613 17478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:53.054677 17478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:53.065070 17491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:53.065171 17496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:53.065093 17493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:53.065223 17478 server_base.cc:1034] running on GCE node
I20260812 02:37:53.065770 17478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:53.066479 17478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:53.067673 17478 hybrid_clock.cc:648] HybridClock initialized: now 1786502273067631 us; error 36 us; skew 500 ppm
Aug 12 02:37:53 dist-test-slave-rmhg krb5kdc[17043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502273, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:53.070785 17478 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:37:53.071920 17478 webserver.cc:460] Webserver started at http://127.3.127.130:38721/ using document root <none> and password file <none>
I20260812 02:37:53.072616 17478 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:53.072701 17478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:53.072896 17478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:53.075230 17478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "7a5ff102979b4558827302e1bd36508d"
format_stamp: "Formatted at 2026-08-12 02:37:53 on dist-test-slave-rmhg"
server_key: "33b9686cfb14175cef6d1508d9508320"
server_key_iv: "1ac218aeef975c32223d029955350eb2"
server_key_version: "encryptionkey@0"
I20260812 02:37:53.075829 17478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "7a5ff102979b4558827302e1bd36508d"
format_stamp: "Formatted at 2026-08-12 02:37:53 on dist-test-slave-rmhg"
server_key: "33b9686cfb14175cef6d1508d9508320"
server_key_iv: "1ac218aeef975c32223d029955350eb2"
server_key_version: "encryptionkey@0"
I20260812 02:37:53.080229 17478 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260812 02:37:53.083186 17508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:53.084673 17478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:37:53.084810 17478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "7a5ff102979b4558827302e1bd36508d"
format_stamp: "Formatted at 2026-08-12 02:37:53 on dist-test-slave-rmhg"
server_key: "33b9686cfb14175cef6d1508d9508320"
server_key_iv: "1ac218aeef975c32223d029955350eb2"
server_key_version: "encryptionkey@0"
I20260812 02:37:53.084939 17478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:53.096980 17478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:53.100198 17478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:53.100400 17478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:53.101065 17478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:53.102035 17478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:53.102087 17478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:53.102134 17478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:53.102164 17478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:53.111807 17478 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:45071
I20260812 02:37:53.111830 17688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:45071 every 8 connection(s)
I20260812 02:37:53.112819 17478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:37:53.119921 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 17478
I20260812 02:37:53.120057 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:37:53.120354 3582 external_mini_cluster.cc:1428] Setting key 19934246d13e3d76c5473f22f37aa90a
Aug 12 02:37:53 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502273, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:53.127282 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:53.114783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:40487 (local address 127.3.127.190:41875)
0812 02:37:53.114939 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:53.114943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:53.115906 (+ 963us) server_negotiation.cc:407] Connection header received
0812 02:37:53.117301 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:53.117304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:53.117365 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:53.117446 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:53.118818 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:53.119358 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:53.120616 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:53.120918 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:53.121009 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:53.123620 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:53.123624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:53.123628 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:53.123663 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:53.125079 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:53.125666 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:53.125669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:53.125670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:53.125757 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:53.126099 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:53.126103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:53.126105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:53.126312 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:53.126404 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:53.126973 (+ 569us) server_negotiation.cc:299] Negotiation successful
0812 02:37:53.127113 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 02:37:53.128198 17694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:53.115216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.130:40487)
0812 02:37:53.115735 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:53.115786 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:53.117043 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:53.117649 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:53.117658 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:53.118120 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:53.118664 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:53.118680 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:53.119483 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:53.119486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:53.120488 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:53.120500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:53.121085 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:53.121089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:53.121316 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:53.121913 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:53.121932 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:53.123433 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:53.125248 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:53.125255 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:53.125274 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:53.125554 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:53.125845 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:53.125848 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:53.125850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:53.126007 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:53.126419 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:53.126427 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:53.126719 (+ 292us) client_negotiation.cc:769] Sending connection context
0812 02:37:53.126907 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 02:37:53.127290 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":133,"threads_started":1}
I20260812 02:37:53.129485 17690 heartbeater.cc:347] Connected to a master server at 127.3.127.190:41875
I20260812 02:37:53.129788 17690 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.127.131@KRBTEST.COM; defaulting to no policy
I20260812 02:37:53.130470 17690 heartbeater.cc:511] Master 127.3.127.190:41875 requested a full tablet report, sending...
I20260812 02:37:53.131644 17121 ts_manager.cc:194] Registered new tserver with Master: 7a5ff102979b4558827302e1bd36508d (127.3.127.130:45071)
I20260812 02:37:53.132233 17121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:40487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
I20260812 02:37:53.152575 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:41875
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:33817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:53.272588 17698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:53.272897 17698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:53.272961 17698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:53.277108 17698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:53.277215 17698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:53.277298 17698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:37:53.282835 17698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:33817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:41875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:53.284034 17698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:53.285171 17698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:53.295689 17708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:53.295712 17698 server_base.cc:1034] running on GCE node
W20260812 02:37:53.295743 17706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:53.295742 17705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:53.296447 17698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:53.297250 17698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:53.298442 17698 hybrid_clock.cc:648] HybridClock initialized: now 1786502273298410 us; error 52 us; skew 500 ppm
Aug 12 02:37:53 dist-test-slave-rmhg krb5kdc[17043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502273, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:53.301425 17698 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:37:53.302390 17698 webserver.cc:460] Webserver started at http://127.3.127.131:38677/ using document root <none> and password file <none>
I20260812 02:37:53.302989 17698 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:53.303062 17698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:53.303270 17698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:37:53.305576 17698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "ece7e8a919fb454c8e0dd720d4456879"
format_stamp: "Formatted at 2026-08-12 02:37:53 on dist-test-slave-rmhg"
server_key: "f5e0ea9c7ec09bd0cab427e5e54abc6e"
server_key_iv: "d007bff55fe1943a7e08b8ab7c0a4362"
server_key_version: "encryptionkey@0"
I20260812 02:37:53.306237 17698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "ece7e8a919fb454c8e0dd720d4456879"
format_stamp: "Formatted at 2026-08-12 02:37:53 on dist-test-slave-rmhg"
server_key: "f5e0ea9c7ec09bd0cab427e5e54abc6e"
server_key_iv: "d007bff55fe1943a7e08b8ab7c0a4362"
server_key_version: "encryptionkey@0"
I20260812 02:37:53.310346 17698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:37:53.313256 17721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:53.314699 17698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:53.314839 17698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "ece7e8a919fb454c8e0dd720d4456879"
format_stamp: "Formatted at 2026-08-12 02:37:53 on dist-test-slave-rmhg"
server_key: "f5e0ea9c7ec09bd0cab427e5e54abc6e"
server_key_iv: "d007bff55fe1943a7e08b8ab7c0a4362"
server_key_version: "encryptionkey@0"
I20260812 02:37:53.314962 17698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:53.338541 17698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:53.342103 17698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:53.342319 17698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:53.342975 17698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:53.343879 17698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:53.343933 17698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:53.343988 17698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:53.344014 17698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:53.353261 17698 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:37111
I20260812 02:37:53.353273 17917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:37111 every 8 connection(s)
I20260812 02:37:53.354254 17698 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:37:53.360870 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 17698
I20260812 02:37:53.361032 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:37:53.361402 3582 external_mini_cluster.cc:1428] Setting key dfcac0b654eab1fae09e0dcfcf609644
Aug 12 02:37:53 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502273, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:37:53.368537 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:53.356139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:42451 (local address 127.3.127.190:41875)
0812 02:37:53.356329 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:53.356334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:53.357272 (+ 938us) server_negotiation.cc:407] Connection header received
0812 02:37:53.358577 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:53.358580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:53.358631 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:53.358714 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:53.359994 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:53.360543 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:53.362109 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:53.362395 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:53.362502 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:53.365355 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:53.365360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:53.365363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:53.365395 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:53.366626 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:53.367132 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:53.367135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:53.367136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:53.367220 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:53.367497 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:53.367499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:53.367500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:53.367711 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:53.367784 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:53.368295 (+ 511us) server_negotiation.cc:299] Negotiation successful
0812 02:37:53.368410 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 02:37:53.369491 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:53.356594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.131:42451)
0812 02:37:53.357076 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:53.357123 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:53.358371 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:53.358901 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:53.358909 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:53.359306 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:53.359850 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:53.359865 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:53.360682 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:53.360687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:53.361998 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:53.362009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:53.362555 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:53.362559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:53.362776 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:53.363427 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:53.363458 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:53.365143 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:37:53.366738 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:53.366744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:53.366759 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:53.367031 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:53.367288 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:37:53.367290 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:37:53.367291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:37:53.367429 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:37:53.367787 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:37:53.367792 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:37:53.368054 (+ 262us) client_negotiation.cc:769] Sending connection context
0812 02:37:53.368219 (+ 165us) client_negotiation.cc:240] Negotiation successful
0812 02:37:53.368505 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":127,"threads_started":1}
I20260812 02:37:53.370690 17918 heartbeater.cc:347] Connected to a master server at 127.3.127.190:41875
I20260812 02:37:53.370916 17918 heartbeater.cc:464] Registering TS with master...
I20260812 02:37:53.371412 17918 heartbeater.cc:511] Master 127.3.127.190:41875 requested a full tablet report, sending...
I20260812 02:37:53.372426 17121 ts_manager.cc:194] Registered new tserver with Master: ece7e8a919fb454c8e0dd720d4456879 (127.3.127.131:37111)
I20260812 02:37:53.372994 17121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:42451
I20260812 02:37:53.375373 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.129.keytab.
Entry for principal oryx/127.3.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.129.keytab.
WARNING: no policy specified for oryx/127.3.127.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.127.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.130.keytab.
Entry for principal oryx/127.3.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.130.keytab.
WARNING: no policy specified for oryx/127.3.127.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.127.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.131.keytab.
Entry for principal oryx/127.3.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.131.keytab.
I20260812 02:37:53.459913 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 17234
I20260812 02:37:53.478703 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 17478
I20260812 02:37:53.494004 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 17698
I20260812 02:37:53.513415 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 17071
I20260812 02:37:53.532667 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:41875
--webserver_interface=127.3.127.190
--webserver_port=45923
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:33817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:41875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:37:53.652559 17933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:53.652868 17933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:53.652908 17933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:53.656929 17933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:37:53.657001 17933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:53.657019 17933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:53.657035 17933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:37:53.657048 17933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:37:53.662626 17933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:33817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:41875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:41875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=45923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:53.663769 17933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:53.664839 17933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:53.674032 17946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:53.674037 17944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:53.674032 17949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:53.674450 17933 server_base.cc:1034] running on GCE node
I20260812 02:37:53.675086 17933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:53.676323 17933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:53.677501 17933 hybrid_clock.cc:648] HybridClock initialized: now 1786502273677480 us; error 32 us; skew 500 ppm
Aug 12 02:37:53 dist-test-slave-rmhg krb5kdc[17043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502273, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:53.680482 17933 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:37:53.681495 17933 webserver.cc:460] Webserver started at http://127.3.127.190:45923/ using document root <none> and password file <none>
I20260812 02:37:53.682052 17933 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:53.682123 17933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:53.686223 17933 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:37:53.688617 17958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:53.690006 17933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 02:37:53.690130 17933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "e47e5630f715400b8b6f537394e88905"
format_stamp: "Formatted at 2026-08-12 02:37:52 on dist-test-slave-rmhg"
server_key: "5c9336c047e39ee0189370f71c99fa7e"
server_key_iv: "11b4dfc586c44188c5fb11bc27835959"
server_key_version: "encryptionkey@0"
I20260812 02:37:53.690724 17933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:53.707648 17933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:53.711244 17933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:53.711432 17933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:53.719345 17933 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:41875
I20260812 02:37:53.719384 18043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:41875 every 8 connection(s)
I20260812 02:37:53.720660 17933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:37:53.726346 18044 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:37:53.728286 18044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905: Bootstrap starting.
I20260812 02:37:53.730783 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 17933
I20260812 02:37:53.731334 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:43613
--local_ip_for_outbound_sockets=127.3.127.129
--tserver_master_addrs=127.3.127.190:41875
--webserver_port=43295
--webserver_interface=127.3.127.129
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:33817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.127.129
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:37:53.747318 18044 log.cc:824] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905: Log is configured to *not* fsync() on all Append() calls
I20260812 02:37:53.754652 18044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:37:53.755051 18044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905: Bootstrap complete.
I20260812 02:37:53.759449 18044 raft_consensus.cc:348] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47e5630f715400b8b6f537394e88905" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41875 } }
I20260812 02:37:53.759902 18044 raft_consensus.cc:729] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47e5630f715400b8b6f537394e88905, State: Initialized, Role: FOLLOWER
I20260812 02:37:53.760464 18044 consensus_queue.cc:260] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47e5630f715400b8b6f537394e88905" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41875 } }
I20260812 02:37:53.760603 18044 raft_consensus.cc:388] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:37:53.760682 18044 raft_consensus.cc:482] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:37:53.760784 18044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [term 1 FOLLOWER]: Advancing to term 2
I20260812 02:37:53.762995 18044 raft_consensus.cc:504] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47e5630f715400b8b6f537394e88905" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41875 } }
I20260812 02:37:53.763374 18044 leader_election.cc:302] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e47e5630f715400b8b6f537394e88905; no voters:
I20260812 02:37:53.763737 18044 leader_election.cc:288] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 02:37:53.763865 18049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 02:37:53.764099 18049 raft_consensus.cc:686] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [term 2 LEADER]: Becoming Leader. State: Replica: e47e5630f715400b8b6f537394e88905, State: Running, Role: LEADER
I20260812 02:37:53.764547 18049 consensus_queue.cc:237] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47e5630f715400b8b6f537394e88905" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41875 } }
I20260812 02:37:53.765249 18044 sys_catalog.cc:564] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:37:53.766240 18052 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e47e5630f715400b8b6f537394e88905. Latest consensus state: current_term: 2 leader_uuid: "e47e5630f715400b8b6f537394e88905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47e5630f715400b8b6f537394e88905" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41875 } } }
I20260812 02:37:53.766271 18050 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e47e5630f715400b8b6f537394e88905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47e5630f715400b8b6f537394e88905" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41875 } } }
I20260812 02:37:53.766386 18052 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:53.766388 18050 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905 [sys.catalog]: This master's current role is: LEADER
I20260812 02:37:53.766808 18066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:37:53.770344 18066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:37:53.771076 18066 catalog_manager.cc:1259] Loaded cluster ID: 189dcfe291e64388b14416b27cc9afcc
I20260812 02:37:53.771133 18066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:37:53.772861 18066 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:37:53.773520 18066 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e47e5630f715400b8b6f537394e88905: Loaded TSK: 0
I20260812 02:37:53.774516 18066 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 02:37:53.866282 18046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:53.866665 18046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:53.866732 18046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:53.871270 18046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:53.871343 18046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:53.871407 18046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:37:53.876931 18046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:33817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.129.keytab
--principal=oryx/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:43613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=43295
--tserver_master_addrs=127.3.127.190:41875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:53.878110 18046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:53.879220 18046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:53.889304 18084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:53.889490 18046 server_base.cc:1034] running on GCE node
W20260812 02:37:53.889287 18088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:53.889581 18093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:53.890190 18046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:53.891038 18046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:53.892248 18046 hybrid_clock.cc:648] HybridClock initialized: now 1786502273892202 us; error 53 us; skew 500 ppm
Aug 12 02:37:53 dist-test-slave-rmhg krb5kdc[17043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502273, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:53.895716 18046 init.cc:377] Logged in from keytab as oryx/127.3.127.129@KRBTEST.COM (short username oryx)
I20260812 02:37:53.896755 18046 webserver.cc:460] Webserver started at http://127.3.127.129:43295/ using document root <none> and password file <none>
I20260812 02:37:53.897447 18046 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:53.897544 18046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:53.902382 18046 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:37:53.905095 18104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:53.906689 18046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:37:53.906836 18046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "1565e6caf697459dbeac744731b13bd5"
format_stamp: "Formatted at 2026-08-12 02:37:52 on dist-test-slave-rmhg"
server_key: "48f20196f0c1c2acddbf6173f2e01825"
server_key_iv: "3361a11e578add1ec4839ce04bdaa249"
server_key_version: "encryptionkey@0"
I20260812 02:37:53.907594 18046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:53.927158 18046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:53.930342 18046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:53.930505 18046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:53.931092 18046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:53.931921 18046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:53.931967 18046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:53.932008 18046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:53.932029 18046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:53.941097 18046 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:43613
I20260812 02:37:53.941120 18277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:43613 every 8 connection(s)
I20260812 02:37:53.942201 18046 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:37:53.950009 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 18046
I20260812 02:37:53.950464 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:45071
--local_ip_for_outbound_sockets=127.3.127.130
--tserver_master_addrs=127.3.127.190:41875
--webserver_port=38721
--webserver_interface=127.3.127.130
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:33817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.127.130
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:37:53 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:53 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
W20260812 02:37:53.956278 18283 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:53.944486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:38471 (local address 127.3.127.190:41875)
0812 02:37:53.945037 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:53.945059 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:53.945307 (+ 248us) server_negotiation.cc:407] Connection header received
0812 02:37:53.946919 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:53.946932 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:53.947291 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:53.947628 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:53.948868 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:53.949907 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:53.951194 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:53.951682 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:53.951832 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:53.955512 (+ 3680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.129:38471: BlockingRecv error: recv got EOF from 127.3.127.129:38471 (error 108)
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":139,"threads_started":1}
I20260812 02:37:53.956367 18284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:53.944610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.129:38471)
0812 02:37:53.945094 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:53.945184 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:53.946463 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:53.947755 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:53.947764 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:53.948193 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:53.948693 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:53.948709 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:53.950045 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:53.950048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:53.951076 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:53.951087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:53.951845 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:53.951849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:53.952034 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:53.952591 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:53.952618 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:53.955281 (+ 2663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":124,"threads_started":1}
W20260812 02:37:53.956976 18278 heartbeater.cc:644] Failed to heartbeat to 127.3.127.190:41875 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.127.190:41875: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Aug 12 02:37:53 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:53 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:37:53.962337 18284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:53.957487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.129:59037)
0812 02:37:53.957591 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:53.957613 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:53.957701 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:53.957922 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:53.957925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:53.958079 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:53.958181 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:53.958187 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:53.958913 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:53.958916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:53.959470 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:53.959479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:53.959909 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:53.959913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:53.960004 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:53.960449 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:53.960461 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:53.962194 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260812 02:37:53.962778 18283 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:53.957596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:59037 (local address 127.3.127.190:41875)
0812 02:37:53.957708 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:53.957712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:53.957728 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:37:53.957773 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:53.957776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:53.957821 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:53.957886 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:53.958292 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:53.958811 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:53.959582 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:53.959804 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:53.959894 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:53.962363 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.129:59037: BlockingRecv error: recv got EOF from 127.3.127.129:59037 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 12 02:37:53 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:53 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:37:53.968900 18284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:53.962915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.129:48463)
0812 02:37:53.963028 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:53.963043 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:53.963133 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:53.963461 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:53.963464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:53.963605 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:53.963690 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:53.963696 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:53.964950 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:53.964953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:53.965568 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:53.965579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:53.966188 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:53.966191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:53.966260 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:53.966818 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:53.966834 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:53.968751 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 02:37:53.969028 18283 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:53.963177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:48463 (local address 127.3.127.190:41875)
0812 02:37:53.963281 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:53.963285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:53.963294 (+ 9us) server_negotiation.cc:407] Connection header received
0812 02:37:53.963325 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:53.963327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:53.963362 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:53.963424 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:53.963828 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:53.964828 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:53.965683 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:53.966074 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:53.966183 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:53.968903 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.129:48463: BlockingRecv error: recv got EOF from 127.3.127.129:48463 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260812 02:37:53.969441 18278 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 02:37:54.072062 18287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:54.072364 18287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:54.072433 18287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:54.076675 18287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:54.076758 18287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:54.076844 18287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:37:54.082551 18287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:33817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.130.keytab
--principal=oryx/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:45071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=38721
--tserver_master_addrs=127.3.127.190:41875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:54.083794 18287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:54.084976 18287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:54.095712 18299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:54.095901 18293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:54.095767 18295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:54.095937 18287 server_base.cc:1034] running on GCE node
I20260812 02:37:54.096498 18287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:54.097296 18287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:54.098474 18287 hybrid_clock.cc:648] HybridClock initialized: now 1786502274098445 us; error 40 us; skew 500 ppm
Aug 12 02:37:54 dist-test-slave-rmhg krb5kdc[17043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502274, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:54.101445 18287 init.cc:377] Logged in from keytab as oryx/127.3.127.130@KRBTEST.COM (short username oryx)
I20260812 02:37:54.102442 18287 webserver.cc:460] Webserver started at http://127.3.127.130:38721/ using document root <none> and password file <none>
I20260812 02:37:54.103061 18287 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:54.103153 18287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:54.107360 18287 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:37:54.109653 18310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:54.110957 18287 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 02:37:54.111080 18287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "7a5ff102979b4558827302e1bd36508d"
format_stamp: "Formatted at 2026-08-12 02:37:53 on dist-test-slave-rmhg"
server_key: "33b9686cfb14175cef6d1508d9508320"
server_key_iv: "1ac218aeef975c32223d029955350eb2"
server_key_version: "encryptionkey@0"
I20260812 02:37:54.111645 18287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:54.125286 18287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:54.128432 18287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:54.128638 18287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:54.129320 18287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:54.130218 18287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:54.130270 18287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:54.130326 18287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:54.130355 18287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:54.139416 18287 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:45071
I20260812 02:37:54.139446 18510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:45071 every 8 connection(s)
I20260812 02:37:54.140846 18287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:37:54.149245 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 18287
I20260812 02:37:54.149715 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:37111
--local_ip_for_outbound_sockets=127.3.127.131
--tserver_master_addrs=127.3.127.190:41875
--webserver_port=38677
--webserver_interface=127.3.127.131
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:33817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.127.131
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:37:54 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:54 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
W20260812 02:37:54.152834 18283 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:54.142925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:57977 (local address 127.3.127.190:41875)
0812 02:37:54.143095 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:54.143099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:54.144072 (+ 973us) server_negotiation.cc:407] Connection header received
0812 02:37:54.145560 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:54.145564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:54.145635 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:54.145714 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:54.147070 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.147618 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.148842 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.149096 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.149219 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:54.152693 (+ 3474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.130:57977: BlockingRecv error: recv got EOF from 127.3.127.130:57977 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 02:37:54.153712 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:54.143375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.130:57977)
0812 02:37:54.143907 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:54.143955 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:54.145317 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:54.145932 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:54.145941 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:54.146342 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:54.146905 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:54.146920 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.147757 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.147760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:54.148706 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:54.148717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.149288 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.149292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:54.149501 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:54.150127 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:54.150146 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:54.152526 (+ 2380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":143,"threads_started":1}
W20260812 02:37:54.154356 18511 heartbeater.cc:644] Failed to heartbeat to 127.3.127.190:41875 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.127.190:41875: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Aug 12 02:37:54 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:54 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:37:54.160601 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:54.154831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.130:43559)
0812 02:37:54.155023 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:54.155038 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:54.155142 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:54.155400 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:54.155402 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:54.155551 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:54.155677 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:54.155683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.156704 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.156706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:54.157327 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:54.157338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.157931 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.157934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:54.158024 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:54.158534 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:54.158546 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:54.160489 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119}
W20260812 02:37:54.161079 18283 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:54.154974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:43559 (local address 127.3.127.190:41875)
0812 02:37:54.155120 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:54.155125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:54.155142 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:37:54.155223 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:54.155226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:54.155280 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:54.155368 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:54.155794 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.156587 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.157438 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.157783 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.157897 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:54.160688 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.130:43559: BlockingRecv error: recv got EOF from 127.3.127.130:43559 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 12 02:37:54 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:54 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:37:54.166240 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:54.161259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.130:35367)
0812 02:37:54.161460 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:54.161472 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:54.161550 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:54.161781 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:54.161784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:54.161933 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:54.162012 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:54.162018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.162729 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.162733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:54.163254 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:54.163261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.163677 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.163681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:54.163757 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:54.164192 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:54.164205 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:54.166107 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133}
W20260812 02:37:54.166361 18283 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:54.161455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:35367 (local address 127.3.127.190:41875)
0812 02:37:54.161584 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:54.161589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:54.161601 (+ 12us) server_negotiation.cc:407] Connection header received
0812 02:37:54.161637 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:54.161640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:54.161683 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:54.161749 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:54.162120 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.162626 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.163362 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.163571 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.163659 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:54.166257 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.130:35367: BlockingRecv error: recv got EOF from 127.3.127.130:35367 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260812 02:37:54.166546 18511 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 02:37:54.272490 18514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:37:54.272846 18514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:37:54.272900 18514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:37:54.277418 18514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:37:54.277498 18514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:37:54.277582 18514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:37:54.283200 18514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:33817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/krb5kdc/oryx_127.3.127.131.keytab
--principal=oryx/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:37111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=38677
--tserver_master_addrs=127.3.127.190:41875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:37:54.284327 18514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:37:54.285462 18514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:37:54.296445 18522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:54.296459 18532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:37:54.296506 18528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:37:54.296737 18514 server_base.cc:1034] running on GCE node
I20260812 02:37:54.297238 18514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:37:54.297957 18514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:37:54.299136 18514 hybrid_clock.cc:648] HybridClock initialized: now 1786502274299118 us; error 50 us; skew 500 ppm
Aug 12 02:37:54 dist-test-slave-rmhg krb5kdc[17043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502274, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:37:54.302172 18514 init.cc:377] Logged in from keytab as oryx/127.3.127.131@KRBTEST.COM (short username oryx)
I20260812 02:37:54.303164 18514 webserver.cc:460] Webserver started at http://127.3.127.131:38677/ using document root <none> and password file <none>
I20260812 02:37:54.303828 18514 fs_manager.cc:359] Metadata directory not provided
I20260812 02:37:54.303922 18514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:37:54.308314 18514 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:37:54.310838 18541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:54.312364 18514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:37:54.312511 18514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "ece7e8a919fb454c8e0dd720d4456879"
format_stamp: "Formatted at 2026-08-12 02:37:53 on dist-test-slave-rmhg"
server_key: "f5e0ea9c7ec09bd0cab427e5e54abc6e"
server_key_iv: "d007bff55fe1943a7e08b8ab7c0a4362"
server_key_version: "encryptionkey@0"
I20260812 02:37:54.313138 18514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:37:54.338250 18514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:37:54.341987 18514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:37:54.342193 18514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:37:54.342844 18514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:37:54.343755 18514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:37:54.343807 18514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:54.343863 18514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:37:54.343887 18514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:37:54.352432 18514 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:37111
I20260812 02:37:54.352496 18731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:37111 every 8 connection(s)
I20260812 02:37:54.353523 18514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:37:54.359125 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 18514
Aug 12 02:37:54 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:54 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
W20260812 02:37:54.366139 18283 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:54.355333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:55883 (local address 127.3.127.190:41875)
0812 02:37:54.355507 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:54.355511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:54.356597 (+ 1086us) server_negotiation.cc:407] Connection header received
0812 02:37:54.358095 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:54.358097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:54.358158 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:54.358222 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:54.359703 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.360435 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.361984 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.362306 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.362391 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:54.366013 (+ 3622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.131:55883: BlockingRecv error: recv got EOF from 127.3.127.131:55883 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 02:37:54.366899 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:54.355798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.131:55883)
0812 02:37:54.356376 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:54.356440 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:54.357865 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:54.358387 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:54.358394 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:54.358771 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:54.359520 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:54.359543 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.360564 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.360567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:54.361858 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:54.361868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.362449 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.362452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:54.362649 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:54.363282 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:54.363305 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:54.365907 (+ 2602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":161,"threads_started":1}
W20260812 02:37:54.367357 18733 heartbeater.cc:644] Failed to heartbeat to 127.3.127.190:41875 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.127.190:41875: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
I20260812 02:37:54.369375 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:54.360180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57600 (local address 127.3.127.190:41875)
0812 02:37:54.360431 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:54.360435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:54.360451 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:37:54.360492 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:54.360494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:54.360551 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:54.360620 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:54.361601 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.362233 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.363326 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.363584 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.363664 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:54.365103 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:37:54.365109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:37:54.365126 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:37:54.365187 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:37:54.366986 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:54.367520 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:54.367526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:54.367533 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:54.367692 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:37:54.368111 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:37:54.368115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:37:54.368117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:37:54.368483 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:37:54.368676 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 02:37:54.368969 (+ 293us) server_negotiation.cc:299] Negotiation successful
0812 02:37:54.369252 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":73,"threads_started":1}
Aug 12 02:37:54 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:54 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:37:54.372740 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:54.367800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.131:53699)
0812 02:37:54.367898 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:54.367911 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:54.367984 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:54.368205 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:54.368207 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:54.368387 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:54.368497 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:54.368502 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.369273 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.369277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:54.369927 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:54.369937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.370426 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.370430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:54.370520 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:54.370925 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:54.370939 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:54.372644 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260812 02:37:54.372854 18283 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:54.367872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:53699 (local address 127.3.127.190:41875)
0812 02:37:54.367975 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:54.367979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:54.367992 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:37:54.368070 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:54.368073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:54.368114 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:54.368175 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:54.368605 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.369122 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.370037 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.370322 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.370402 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:54.372764 (+ 2362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.131:53699: BlockingRecv error: recv got EOF from 127.3.127.131:53699 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Aug 12 02:37:54 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:54 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:37:54.378315 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:54.373246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.131:46073)
0812 02:37:54.373349 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:54.373363 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:54.373436 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:54.373631 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:54.373634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:54.373773 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:54.373841 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:54.373846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.374509 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.374512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:54.375019 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:54.375026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.375647 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.375651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:54.375751 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:54.376183 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:54.376196 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:54.378176 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260812 02:37:54.378571 18283 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:54.373278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:46073 (local address 127.3.127.190:41875)
0812 02:37:54.373400 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:54.373404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:54.373417 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:37:54.373504 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:54.373507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:54.373544 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:54.373608 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:54.373937 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.374414 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.375119 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.375428 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.375525 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:54.378320 (+ 2795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.131:46073: BlockingRecv error: recv got EOF from 127.3.127.131:46073 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260812 02:37:54.378861 18733 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 12 02:37:54 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:54 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:37:54.976083 18739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:54.970193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.129:36155)
0812 02:37:54.970475 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:54.970490 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:54.970603 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:54.970855 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:54.970857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:54.971038 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:54.971170 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:54.971177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.971908 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.971910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:54.972498 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:54.972508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.973105 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.973110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:54.973262 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:54.973853 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:54.973866 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:54.975951 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":80,"threads_started":1}
W20260812 02:37:54.976423 18742 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:54.970323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:36155 (local address 127.3.127.190:41875)
0812 02:37:54.970589 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:54.970593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:54.970609 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:37:54.970694 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:54.970696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:54.970742 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:54.970832 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:54.971269 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.971814 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.972668 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:54.972921 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:54.973017 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:54.976286 (+ 3269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.129:36155: BlockingRecv error: recv got EOF from 127.3.127.129:36155 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":97,"threads_started":1}
Aug 12 02:37:55 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:55 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:37:55.173287 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:55.167395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.130:36155)
0812 02:37:55.167727 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:55.167745 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:55.167876 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:55.168167 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:55.168170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:55.168406 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:55.168608 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:55.168616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:55.169565 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:55.169568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:55.170224 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:55.170236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:55.170682 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:55.170684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:55.170752 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:55.171286 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:55.171300 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:55.173108 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":107,"threads_started":1}
W20260812 02:37:55.173482 18742 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:55.167522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:36155 (local address 127.3.127.190:41875)
0812 02:37:55.167732 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:55.167737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:55.167823 (+ 86us) server_negotiation.cc:407] Connection header received
0812 02:37:55.167970 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:55.167973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:55.168028 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:55.168113 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:55.168734 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:55.169439 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:55.170336 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:55.170562 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:55.170641 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:55.173339 (+ 2698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.130:36155: BlockingRecv error: recv got EOF from 127.3.127.130:36155 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Aug 12 02:37:55 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:55 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:37:55.385028 18747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:55.379503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.131:57837)
0812 02:37:55.379832 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:55.379844 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:55.379963 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:55.380225 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:55.380228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:55.380440 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:55.380575 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:55.380582 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:55.381437 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:55.381441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:55.382033 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:55.382042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:55.382489 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:55.382492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:55.382589 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:55.383073 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:55.383084 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:55.384904 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":92,"threads_started":1}
W20260812 02:37:55.385227 18742 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:55.379593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:57837 (local address 127.3.127.190:41875)
0812 02:37:55.379728 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:55.379734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:55.379925 (+ 191us) server_negotiation.cc:407] Connection header received
0812 02:37:55.380055 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:55.380058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:55.380109 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:55.380205 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:55.380696 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:55.381335 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:55.382147 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:55.382385 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:55.382473 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:55.385061 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.131:57837: BlockingRecv error: recv got EOF from 127.3.127.131:57837 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 12 02:37:55 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:55 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:37:55.983320 18748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:55.977123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.129:60497)
0812 02:37:55.977487 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:55.977504 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:55.977626 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:55.977952 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:55.977955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:55.978151 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:55.978271 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:55.978277 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:55.979254 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:55.979257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:55.980007 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:55.980021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:55.980489 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:55.980492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:55.980557 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:55.981200 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:55.981218 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:55.983175 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":92,"threads_started":1}
W20260812 02:37:55.983448 18749 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:55.977287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:60497 (local address 127.3.127.190:41875)
0812 02:37:55.977597 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:55.977600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:55.977618 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:37:55.977719 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:55.977722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:55.977772 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:55.977874 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:55.978411 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:55.979114 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:55.980149 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:55.980391 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:55.980475 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:55.983312 (+ 2837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.129:60497: BlockingRecv error: recv got EOF from 127.3.127.129:60497 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":92,"threads_started":1}
Aug 12 02:37:56 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:56 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:37:56.182442 18750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:56.174117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.130:49575)
0812 02:37:56.174408 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:56.174423 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:56.174534 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:56.174744 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:56.174746 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:56.174925 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:56.175031 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:56.175037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:56.175877 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:56.175881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:56.176539 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:56.176552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:56.177033 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:56.177035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:56.177102 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:56.180358 (+ 3256us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:56.180373 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:56.182282 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":102,"threads_started":1}
W20260812 02:37:56.182593 18749 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:56.174211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:49575 (local address 127.3.127.190:41875)
0812 02:37:56.174375 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:56.174380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:56.174521 (+ 141us) server_negotiation.cc:407] Connection header received
0812 02:37:56.174605 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:56.174607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:56.174654 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:56.174720 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:56.175164 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:56.175744 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:56.176686 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:56.176926 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:56.177019 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:56.182463 (+ 5444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.130:49575: BlockingRecv error: recv got EOF from 127.3.127.130:49575 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 12 02:37:56 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:56 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:37:56.391043 18751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:56.385871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.131:33165)
0812 02:37:56.386222 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:56.386237 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:56.386351 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:56.386585 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:56.386587 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:56.386805 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:56.386915 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:56.386921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:56.387612 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:56.387614 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:56.388192 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:56.388201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:56.388585 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:56.388588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:56.388683 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:56.389127 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:56.389138 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:56.390869 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":114,"threads_started":1}
W20260812 02:37:56.391129 18749 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:56.386070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:33165 (local address 127.3.127.190:41875)
0812 02:37:56.386222 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:56.386226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:56.386293 (+ 67us) server_negotiation.cc:407] Connection header received
0812 02:37:56.386439 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:56.386441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:56.386493 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:56.386553 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:56.387009 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:56.387532 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:56.388280 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:56.388488 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:56.388555 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:56.391014 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.131:33165: BlockingRecv error: recv got EOF from 127.3.127.131:33165 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 12 02:37:56 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:56 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:37:56.990056 18752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:56.984193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.129:59921)
0812 02:37:56.984484 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:56.984497 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:56.984638 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:56.984938 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:56.984941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:56.985139 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:56.985292 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:56.985298 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:56.986199 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:56.986202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:56.986845 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:56.986855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:56.987323 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:56.987326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:56.987390 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:56.987953 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:56.987965 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:56.989868 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":82,"threads_started":1}
W20260812 02:37:56.990170 18760 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:56.984341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:59921 (local address 127.3.127.190:41875)
0812 02:37:56.984600 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:56.984604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:56.984619 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:37:56.984724 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:56.984726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:56.984774 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:56.984860 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:56.985461 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:56.986054 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:56.986978 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:56.987222 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:56.987307 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:56.990028 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.129:59921: BlockingRecv error: recv got EOF from 127.3.127.129:59921 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":100,"threads_started":1}
Aug 12 02:37:57 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:57 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:37:57.189215 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:57.183454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.130:50989)
0812 02:37:57.183751 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:57.183765 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:57.183887 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:57.184113 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:57.184115 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:57.184309 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:57.184426 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:57.184431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:57.185408 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:57.185413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:57.186090 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:57.186105 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:57.186573 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:57.186577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:57.186673 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:57.187155 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:57.187168 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:57.189035 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":82,"threads_started":1}
W20260812 02:37:57.189345 18760 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:57.183580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:50989 (local address 127.3.127.190:41875)
0812 02:37:57.183771 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:57.183775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:57.183839 (+ 64us) server_negotiation.cc:407] Connection header received
0812 02:37:57.183964 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:57.183966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:57.184014 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:57.184084 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:57.184589 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:57.185259 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:57.186211 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:57.186460 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:57.186554 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:57.189224 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.130:50989: BlockingRecv error: recv got EOF from 127.3.127.130:50989 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 12 02:37:57 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:57 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:37:57.397958 18765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:57.391980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.131:43367)
0812 02:37:57.392320 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:57.392333 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:57.392444 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:57.392670 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:57.392672 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:57.392853 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:57.392957 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:57.392962 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:57.393824 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:57.393828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:57.394534 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:57.394547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:57.395005 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:57.395008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:57.395079 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:57.395600 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:57.395615 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:57.397812 (+ 2197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":86,"threads_started":1}
W20260812 02:37:57.398198 18760 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:57.392032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:43367 (local address 127.3.127.190:41875)
0812 02:37:57.392246 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:57.392250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:57.392404 (+ 154us) server_negotiation.cc:407] Connection header received
0812 02:37:57.392518 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:57.392521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:57.392571 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:57.392647 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:57.393114 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:57.393703 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:57.394669 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:57.394907 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:57.395008 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:57.398066 (+ 3058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.131:43367: BlockingRecv error: recv got EOF from 127.3.127.131:43367 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 12 02:37:57 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:57 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:37:57.997051 18768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:57.991108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.129:50557)
0812 02:37:57.991414 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:57.991429 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:57.991529 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:57.991782 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:57.991784 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:57.991971 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:57.992083 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:57.992088 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:57.992905 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:57.992907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:57.993522 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:57.993533 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:57.994037 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:57.994042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:57.994115 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:57.994627 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:57.994640 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:57.996889 (+ 2249us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":105,"threads_started":1}
W20260812 02:37:57.997232 18769 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:57.991249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:50557 (local address 127.3.127.190:41875)
0812 02:37:57.991498 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:57.991502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:57.991519 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:37:57.991609 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:57.991612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:57.991661 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:57.991751 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:57.992250 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:57.992812 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:57.993645 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:57.993912 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:57.993997 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:57.997048 (+ 3051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.129:50557: BlockingRecv error: recv got EOF from 127.3.127.129:50557 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
Aug 12 02:37:58 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:58 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:37:58.195847 18773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:58.190032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.130:47933)
0812 02:37:58.190360 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:58.190373 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:58.190499 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:58.190764 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:58.190766 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:58.190950 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:58.191063 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:58.191068 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:58.191981 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:58.191984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:58.192576 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:58.192622 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:58.193102 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:58.193106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:58.193216 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:58.193761 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:58.193775 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:58.195711 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":120,"threads_started":1}
W20260812 02:37:58.196010 18769 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:58.190109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:47933 (local address 127.3.127.190:41875)
0812 02:37:58.190240 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:58.190245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:58.190458 (+ 213us) server_negotiation.cc:407] Connection header received
0812 02:37:58.190599 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:58.190603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:58.190661 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:58.190727 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:58.191205 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:58.191849 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:58.192740 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:58.192975 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:58.193063 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:58.195878 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.130:47933: BlockingRecv error: recv got EOF from 127.3.127.130:47933 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 12 02:37:58 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:58 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:37:58.404511 18776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:58.398962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.131:51735)
0812 02:37:58.399251 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:58.399265 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:58.399385 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:58.399627 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:58.399630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:58.399861 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:58.399969 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:58.399975 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:58.400732 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:58.400736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:58.401538 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:58.401550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:58.401985 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:58.401989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:58.402073 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:58.402545 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:58.402557 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:58.404338 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":90,"threads_started":1}
W20260812 02:37:58.404613 18769 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:58.399048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:51735 (local address 127.3.127.190:41875)
0812 02:37:58.399215 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:58.399220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:58.399332 (+ 112us) server_negotiation.cc:407] Connection header received
0812 02:37:58.399482 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:58.399484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:58.399531 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:58.399591 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:58.400084 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:58.400623 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:58.401652 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:58.401879 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:58.401968 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:58.404507 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.131:51735: BlockingRecv error: recv got EOF from 127.3.127.131:51735 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 12 02:37:59 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:59 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:37:59.003943 18777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:58.998055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.129:51283)
0812 02:37:58.998433 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:58.998450 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:58.998581 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:58.998863 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:58.998866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:58.999078 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:58.999191 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:58.999198 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:58.999992 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:58.999996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:59.000609 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:59.000619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:59.001259 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:59.001263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:59.001358 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:59.001910 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:59.001923 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:59.003787 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":127,"threads_started":1}
W20260812 02:37:59.004063 18779 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:58.998150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:51283 (local address 127.3.127.190:41875)
0812 02:37:58.998434 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:58.998439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:58.998533 (+ 94us) server_negotiation.cc:407] Connection header received
0812 02:37:58.998675 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:58.998678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:58.998738 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:58.998821 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:58.999326 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:58.999882 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:59.000761 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:59.001094 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:59.001247 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:59.003943 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.129:51283: BlockingRecv error: recv got EOF from 127.3.127.129:51283 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":102,"threads_started":1}
Aug 12 02:37:59 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:59 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:37:59.202183 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:59.196643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.130:34029)
0812 02:37:59.196922 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:59.196936 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:59.197040 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:59.197319 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:59.197322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:59.197517 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:59.197627 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:59.197633 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:59.198415 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:59.198419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:59.199148 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:59.199162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:59.199592 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:59.199595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:59.199682 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:59.200172 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:59.200186 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:59.202048 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":78,"threads_started":1}
W20260812 02:37:59.202302 18779 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:59.196791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:34029 (local address 127.3.127.190:41875)
0812 02:37:59.196928 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:59.196932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:59.197005 (+ 73us) server_negotiation.cc:407] Connection header received
0812 02:37:59.197119 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:59.197122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:59.197206 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:59.197294 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:59.197761 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:59.198304 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:59.199265 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:59.199502 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:59.199582 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:59.202197 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.130:34029: BlockingRecv error: recv got EOF from 127.3.127.130:34029 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 12 02:37:59 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:37:59 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:37:59.410737 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:37:59.405525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.131:36365)
0812 02:37:59.405842 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0812 02:37:59.405857 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:37:59.405968 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:37:59.406202 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:37:59.406204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:37:59.406396 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:37:59.406500 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:59.406505 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:59.407239 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:59.407242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:59.407816 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:37:59.407823 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:37:59.408233 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:37:59.408235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:37:59.408297 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:59.408766 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:37:59.408777 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:37:59.410615 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":113,"threads_started":1}
W20260812 02:37:59.410897 18779 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:37:59.405612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:36365 (local address 127.3.127.190:41875)
0812 02:37:59.405767 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 02:37:59.405772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:37:59.405927 (+ 155us) server_negotiation.cc:407] Connection header received
0812 02:37:59.406040 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:37:59.406042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:37:59.406092 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:37:59.406162 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:37:59.406600 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:59.407152 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:59.407909 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:37:59.408146 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:37:59.408223 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:37:59.410774 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.131:36365: BlockingRecv error: recv got EOF from 127.3.127.131:36365 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 12 02:38:00 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:00 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:00.010368 18785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:00.004821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.129:55137)
0812 02:38:00.005145 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:00.005204 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:00.005318 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:00.005542 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:00.005545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:00.005746 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:00.005854 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:00.005860 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:00.006657 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:00.006660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:00.007254 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:00.007263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:00.007659 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:00.007662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:00.007730 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:00.008259 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:00.008272 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:00.010200 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":97,"threads_started":1}
W20260812 02:38:00.010511 18787 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:00.004914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:55137 (local address 127.3.127.190:41875)
0812 02:38:00.005222 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:00.005226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:00.005272 (+ 46us) server_negotiation.cc:407] Connection header received
0812 02:38:00.005392 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:00.005394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:00.005439 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:00.005516 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:00.005972 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:00.006540 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:00.007345 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:00.007571 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:00.007642 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:00.010385 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.129:55137: BlockingRecv error: recv got EOF from 127.3.127.129:55137 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":110,"threads_started":1}
Aug 12 02:38:00 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:00 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:00.211526 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:00.203117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.130:38595)
0812 02:38:00.203437 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:00.203452 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:00.203571 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:00.203807 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:00.203809 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:00.204008 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:00.204116 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:00.204122 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:00.204853 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:00.204855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:00.205449 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:00.205458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:00.205867 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:00.205869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:00.205929 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:00.209480 (+ 3551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:00.209495 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:00.211393 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":104,"threads_started":1}
W20260812 02:38:00.211676 18787 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:00.203243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:38595 (local address 127.3.127.190:41875)
0812 02:38:00.203414 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:00.203417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:00.203526 (+ 109us) server_negotiation.cc:407] Connection header received
0812 02:38:00.203650 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:00.203653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:00.203703 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:00.203773 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:00.204213 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:00.204766 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:00.205546 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:00.205776 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:00.205855 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:00.211561 (+ 5706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.130:38595: BlockingRecv error: recv got EOF from 127.3.127.130:38595 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 12 02:38:00 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:00 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:00.417670 18790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:00.411692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.131:59135)
0812 02:38:00.411957 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:00.411972 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:00.412098 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:00.412399 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:00.412403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:00.412636 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:00.412838 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:00.412847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:00.413781 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:00.413784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:00.414385 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:00.414395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:00.414819 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:00.414821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:00.414880 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:00.415398 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:00.415410 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:00.417545 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":83,"threads_started":1}
W20260812 02:38:00.417963 18787 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:00.411822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:59135 (local address 127.3.127.190:41875)
0812 02:38:00.411962 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:00.411965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:00.412063 (+ 98us) server_negotiation.cc:407] Connection header received
0812 02:38:00.412205 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:00.412209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:00.412267 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:00.412344 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:00.412950 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:00.413673 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:00.414500 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:00.414731 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:00.414812 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:00.417773 (+ 2961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.131:59135: BlockingRecv error: recv got EOF from 127.3.127.131:59135 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 12 02:38:01 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:01 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:01.017302 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:01.011398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.129:38615)
0812 02:38:01.011833 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:01.011849 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:01.011971 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:01.012207 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:01.012209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:01.012387 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:01.012489 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:01.012495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:01.013321 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:01.013324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:01.014066 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:01.014080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:01.014619 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:01.014623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:01.014712 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:01.015253 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:01.015266 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:01.017107 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":93,"threads_started":1}
W20260812 02:38:01.017478 18793 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:01.011550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:38615 (local address 127.3.127.190:41875)
0812 02:38:01.011852 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:01.011855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:01.011929 (+ 74us) server_negotiation.cc:407] Connection header received
0812 02:38:01.012044 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:01.012047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:01.012094 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:01.012198 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:01.012579 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:01.013143 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:01.014202 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:01.014472 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:01.014567 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:01.017333 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.129:38615: BlockingRecv error: recv got EOF from 127.3.127.129:38615 (error 108)
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":65,"threads_started":1}
Aug 12 02:38:01 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:01 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:01.223587 18798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:01.212407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.130:59973)
0812 02:38:01.212755 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:01.212768 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:01.212889 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:01.213227 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:01.213229 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:01.213414 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:01.213531 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:01.213538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:01.214360 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:01.214363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:01.215019 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:01.215031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:01.215493 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:01.215496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:01.215565 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:01.221091 (+ 5526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:01.221114 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:01.223365 (+ 2251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":88,"threads_started":1}
W20260812 02:38:01.223737 18793 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:01.212490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:59973 (local address 127.3.127.190:41875)
0812 02:38:01.212638 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:01.212643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:01.212859 (+ 216us) server_negotiation.cc:407] Connection header received
0812 02:38:01.212991 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:01.212994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:01.213043 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:01.213123 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:01.213637 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:01.214227 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:01.215141 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:01.215383 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:01.215467 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:01.223623 (+ 8156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.130:59973: BlockingRecv error: recv got EOF from 127.3.127.130:59973 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 12 02:38:01 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:01 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:01.423967 18799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:01.418436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.131:43243)
0812 02:38:01.418726 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:01.418739 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:01.418855 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:01.419093 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:01.419095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:01.419310 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:01.419431 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:01.419438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:01.420220 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:01.420224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:01.420787 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:01.420797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:01.421265 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:01.421268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:01.421351 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:01.421915 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:01.421927 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:01.423840 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":70,"threads_started":1}
W20260812 02:38:01.424124 18793 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:01.418489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:43243 (local address 127.3.127.190:41875)
0812 02:38:01.418634 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:01.418639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:01.418809 (+ 170us) server_negotiation.cc:407] Connection header received
0812 02:38:01.418941 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:01.418945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:01.418996 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:01.419061 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:01.419549 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:01.420092 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:01.420902 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:01.421128 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:01.421244 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:01.424001 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.131:43243: BlockingRecv error: recv got EOF from 127.3.127.131:43243 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 12 02:38:02 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:02 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:02.023944 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:02.018060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.129:34901)
0812 02:38:02.018400 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:02.018414 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:02.018513 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:02.018847 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:02.018849 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:02.019038 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:02.019147 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:02.019153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:02.020009 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:02.020012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:02.020636 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:02.020647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:02.021127 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:02.021130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:02.021252 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:02.021727 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:02.021738 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:02.023813 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":92,"threads_started":1}
W20260812 02:38:02.024123 18802 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:02.018201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:34901 (local address 127.3.127.190:41875)
0812 02:38:02.018441 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:02.018445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:02.018460 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:38:02.018639 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:02.018642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:02.018697 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:02.018768 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:02.019299 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:02.019867 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:02.020762 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:02.020997 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:02.021097 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:02.023978 (+ 2881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.129:34901: BlockingRecv error: recv got EOF from 127.3.127.129:34901 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":89,"threads_started":1}
Aug 12 02:38:02 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:02 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:02.230798 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:02.224542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.130:36617)
0812 02:38:02.224969 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:02.224983 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:02.225092 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:02.225361 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:02.225363 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:02.225550 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:02.225712 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:02.225718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:02.226718 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:02.226721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:02.227300 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:02.227309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:02.227817 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:02.227820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:02.227888 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:02.228439 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:02.228453 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:02.230663 (+ 2210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":79,"threads_started":1}
W20260812 02:38:02.231086 18802 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:02.224662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:36617 (local address 127.3.127.190:41875)
0812 02:38:02.224818 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:02.224823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:02.225067 (+ 244us) server_negotiation.cc:407] Connection header received
0812 02:38:02.225192 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:02.225195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:02.225254 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:02.225329 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:02.225847 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:02.226591 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:02.227431 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:02.227691 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:02.227783 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:02.230869 (+ 3086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.130:36617: BlockingRecv error: recv got EOF from 127.3.127.130:36617 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 12 02:38:02 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:02 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:02.431051 18804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:02.424981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.131:60791)
0812 02:38:02.425308 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:02.425322 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:02.425444 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:02.425771 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:02.425774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:02.425957 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:02.426083 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:02.426089 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:02.426874 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:02.426876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:02.427562 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:02.427575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:02.428147 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:02.428151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:02.428248 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:02.428938 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:02.428952 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:02.430915 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":81,"threads_started":1}
W20260812 02:38:02.431288 18802 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:02.425144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:60791 (local address 127.3.127.190:41875)
0812 02:38:02.425324 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:02.425328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:02.425392 (+ 64us) server_negotiation.cc:407] Connection header received
0812 02:38:02.425591 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:02.425594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:02.425651 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:02.425731 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:02.426197 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:02.426771 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:02.427682 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:02.427993 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:02.428095 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:02.431143 (+ 3048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.131:60791: BlockingRecv error: recv got EOF from 127.3.127.131:60791 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 12 02:38:03 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:03 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:03.030359 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:03.024857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.129:39023)
0812 02:38:03.025201 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:03.025220 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:03.025329 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:03.025554 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:03.025556 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:03.025739 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:03.025843 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:03.025848 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:03.026594 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:03.026597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:03.027177 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:03.027185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:03.027633 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:03.027636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:03.027709 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:03.028247 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:03.028262 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:03.030219 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":77,"threads_started":1}
W20260812 02:38:03.030506 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:03.024991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:39023 (local address 127.3.127.190:41875)
0812 02:38:03.025302 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:03.025305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:03.025320 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:38:03.025400 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:03.025402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:03.025449 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:03.025526 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:03.025932 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:03.026491 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:03.027286 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:03.027529 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:03.027604 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:03.030380 (+ 2776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.129:39023: BlockingRecv error: recv got EOF from 127.3.127.129:39023 (error 108)
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":143,"threads_started":1}
Aug 12 02:38:03 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:03 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:03.237710 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:03.231778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.130:33953)
0812 02:38:03.232077 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:03.232094 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:03.232210 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:03.232539 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:03.232541 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:03.232760 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:03.232873 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:03.232878 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:03.233717 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:03.233721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:03.234315 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:03.234326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:03.234794 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:03.234802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:03.234883 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:03.235497 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:03.235512 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:03.237548 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":105,"threads_started":1}
W20260812 02:38:03.238001 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:03.231906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:33953 (local address 127.3.127.190:41875)
0812 02:38:03.232078 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:03.232082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:03.232160 (+ 78us) server_negotiation.cc:407] Connection header received
0812 02:38:03.232340 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:03.232343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:03.232402 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:03.232494 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:03.233014 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:03.233592 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:03.234419 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:03.234641 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:03.234735 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:03.237764 (+ 3029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.130:33953: BlockingRecv error: recv got EOF from 127.3.127.130:33953 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 12 02:38:03 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:03 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:03.438215 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:03.432061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.131:59697)
0812 02:38:03.432387 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:03.432402 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:03.432519 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:03.432789 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:03.432792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:03.433009 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:03.433183 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:03.433192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:03.434112 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:03.434115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:03.434786 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:03.434798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:03.435427 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:03.435431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:03.435551 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:03.436127 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:03.436140 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:03.438071 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":90,"threads_started":1}
W20260812 02:38:03.438370 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:03.432167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:59697 (local address 127.3.127.190:41875)
0812 02:38:03.432304 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:03.432308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:03.432483 (+ 175us) server_negotiation.cc:407] Connection header received
0812 02:38:03.432614 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:03.432617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:03.432679 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:03.432741 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:03.433331 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:03.433994 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:03.434923 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:03.435272 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:03.435395 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:03.438253 (+ 2858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.131:59697: BlockingRecv error: recv got EOF from 127.3.127.131:59697 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 12 02:38:04 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:04 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:04.037067 18814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:04.031443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.129:47039)
0812 02:38:04.031760 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:04.031774 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:04.031872 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:04.032120 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:04.032122 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:04.032308 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:04.032416 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:04.032422 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:04.033260 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:04.033263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:04.033857 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:04.033867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:04.034404 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:04.034408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:04.034496 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:04.035019 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:04.035040 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:04.036940 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":84,"threads_started":1}
W20260812 02:38:04.037266 18815 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:04.031549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:47039 (local address 127.3.127.190:41875)
0812 02:38:04.031811 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:04.031815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:04.031831 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:38:04.031955 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:04.031958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:04.032004 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:04.032095 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:04.032519 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:04.033076 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:04.034006 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:04.034278 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:04.034377 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:04.037091 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.129:47039: BlockingRecv error: recv got EOF from 127.3.127.129:47039 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":107,"threads_started":1}
Aug 12 02:38:04 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:04 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:04.244059 18820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:04.238553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.130:59635)
0812 02:38:04.238917 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:04.238930 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:04.239037 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:04.239291 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:04.239294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:04.239493 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:04.239594 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:04.239600 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:04.240385 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:04.240388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:04.240968 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:04.240977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:04.241519 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:04.241523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:04.241607 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:04.242113 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:04.242126 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:04.243898 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":121,"threads_started":1}
W20260812 02:38:04.244192 18815 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:04.238602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:59635 (local address 127.3.127.190:41875)
0812 02:38:04.238762 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:04.238767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:04.239006 (+ 239us) server_negotiation.cc:407] Connection header received
0812 02:38:04.239125 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:04.239128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:04.239182 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:04.239248 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:04.239730 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:04.240269 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:04.241084 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:04.241405 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:04.241499 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:04.244086 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.130:59635: BlockingRecv error: recv got EOF from 127.3.127.130:59635 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 12 02:38:04 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:04 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:04.445178 18821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:04.439298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.131:38605)
0812 02:38:04.439586 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:04.439600 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:04.439703 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:04.439925 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:04.439927 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:04.440133 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:04.440239 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:04.440245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:04.441052 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:04.441056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:04.441730 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:04.441742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:04.442184 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:04.442188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:04.442258 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:04.442788 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:04.442800 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:04.445012 (+ 2212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":91,"threads_started":1}
W20260812 02:38:04.445340 18815 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:04.439468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:38605 (local address 127.3.127.190:41875)
0812 02:38:04.439602 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:04.439606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:04.439663 (+ 57us) server_negotiation.cc:407] Connection header received
0812 02:38:04.439773 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:04.439776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:04.439821 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:04.439895 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:04.440348 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:04.440900 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:04.441843 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:04.442077 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:04.442176 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:04.445222 (+ 3046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.131:38605: BlockingRecv error: recv got EOF from 127.3.127.131:38605 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 12 02:38:05 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:05 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:05.043308 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:05.037885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.129:50515)
0812 02:38:05.038180 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:05.038194 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:05.038299 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:05.038516 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:05.038518 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:05.038713 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:05.038822 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:05.038829 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:05.039570 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:05.039572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:05.040138 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:05.040147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:05.040579 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:05.040581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:05.040640 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:05.041145 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:05.041182 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:05.043148 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
W20260812 02:38:05.043449 18823 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:05.037952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:50515 (local address 127.3.127.190:41875)
0812 02:38:05.038198 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:05.038202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:05.038258 (+ 56us) server_negotiation.cc:407] Connection header received
0812 02:38:05.038370 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:05.038372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:05.038416 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:05.038492 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:05.038916 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:05.039483 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:05.040246 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:05.040486 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:05.040559 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:05.043328 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.129:50515: BlockingRecv error: recv got EOF from 127.3.127.129:50515 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
Aug 12 02:38:05 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:05 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:05.250690 18827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:05.245050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.130:52565)
0812 02:38:05.245381 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:05.245397 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:05.245528 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:05.245787 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:05.245790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:05.246007 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:05.246113 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:05.246120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:05.246928 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:05.246932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:05.247546 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:05.247556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:05.248005 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:05.248009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:05.248098 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:05.248605 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:05.248620 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:05.250501 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":125,"threads_started":1}
W20260812 02:38:05.250772 18823 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:05.245218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:52565 (local address 127.3.127.190:41875)
0812 02:38:05.245384 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:05.245389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:05.245474 (+ 85us) server_negotiation.cc:407] Connection header received
0812 02:38:05.245625 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:05.245628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:05.245677 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:05.245750 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:05.246238 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:05.246816 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:05.247667 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:05.247891 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:05.247981 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:05.250669 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.130:52565: BlockingRecv error: recv got EOF from 127.3.127.130:52565 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 12 02:38:05 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:05 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:05.451689 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:05.446108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.131:54589)
0812 02:38:05.446414 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:05.446430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:05.446563 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:05.446850 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:05.446854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:05.447082 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:05.447222 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:05.447229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:05.447957 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:05.447959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:05.448537 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:05.448544 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:05.448980 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:05.448982 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:05.449046 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:05.449620 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:05.449633 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:05.451531 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":84,"threads_started":1}
W20260812 02:38:05.451843 18823 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:05.446215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:54589 (local address 127.3.127.190:41875)
0812 02:38:05.446356 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:05.446362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:05.446506 (+ 144us) server_negotiation.cc:407] Connection header received
0812 02:38:05.446654 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:05.446657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:05.446717 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:05.446792 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:05.447332 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:05.447869 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:05.448655 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:05.448889 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:05.448972 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:05.451738 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.131:54589: BlockingRecv error: recv got EOF from 127.3.127.131:54589 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 12 02:38:06 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:06 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:06.050071 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:06.044275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.129:49501)
0812 02:38:06.044603 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:06.044621 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:06.044746 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:06.045001 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:06.045003 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:06.045259 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:06.045376 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:06.045382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:06.046212 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:06.046215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:06.046946 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:06.046959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:06.047438 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:06.047440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:06.047502 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:06.048014 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:06.048027 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:06.049941 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":108,"threads_started":1}
W20260812 02:38:06.050284 18832 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:06.044373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:49501 (local address 127.3.127.190:41875)
0812 02:38:06.044683 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:06.044686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:06.044704 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:38:06.044835 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:06.044837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:06.044887 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:06.044978 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:06.045517 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:06.046085 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:06.047078 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:06.047336 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:06.047421 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:06.050135 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.129:49501: BlockingRecv error: recv got EOF from 127.3.127.129:49501 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":90,"threads_started":1}
Aug 12 02:38:06 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:06 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:06.261885 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:06.251778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.130:60515)
0812 02:38:06.252108 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:06.252124 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:06.252360 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:06.252696 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:06.252699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:06.252965 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:06.253104 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:06.253112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:06.253914 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:06.253917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:06.254544 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:06.254554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:06.254990 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:06.254993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:06.255075 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:06.259461 (+ 4386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:06.259481 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:06.261705 (+ 2224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":95,"threads_started":1}
W20260812 02:38:06.262023 18832 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:06.251882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:60515 (local address 127.3.127.190:41875)
0812 02:38:06.252049 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:06.252053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:06.252214 (+ 161us) server_negotiation.cc:407] Connection header received
0812 02:38:06.252503 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:06.252505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:06.252557 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:06.252648 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:06.253236 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:06.253784 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:06.254644 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:06.254869 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:06.254940 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:06.261908 (+ 6968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.130:60515: BlockingRecv error: recv got EOF from 127.3.127.130:60515 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 12 02:38:06 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:06 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:06.458640 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:06.452687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.131:36699)
0812 02:38:06.453028 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:06.453047 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:06.453219 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:06.453480 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:06.453482 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:06.453700 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:06.453806 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:06.453813 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:06.454582 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:06.454586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:06.455146 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:06.455156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:06.455625 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:06.455629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:06.455718 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:06.456255 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:06.456268 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:06.458484 (+ 2216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":131,"threads_started":1}
W20260812 02:38:06.458969 18832 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:06.452847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:36699 (local address 127.3.127.190:41875)
0812 02:38:06.453028 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:06.453057 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:06.453126 (+ 69us) server_negotiation.cc:407] Connection header received
0812 02:38:06.453317 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:06.453321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:06.453373 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:06.453442 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:06.453932 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:06.454471 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:06.455275 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:06.455513 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:06.455606 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:06.458723 (+ 3117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.131:36699: BlockingRecv error: recv got EOF from 127.3.127.131:36699 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 12 02:38:07 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:07 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:07.057135 18836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:07.051082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.129:44463)
0812 02:38:07.051450 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:07.051478 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:07.051621 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:07.052039 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:07.052042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:07.052287 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:07.052430 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:07.052440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:07.053488 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:07.053490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:07.054122 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:07.054132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:07.054552 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:07.054554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:07.054616 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:07.055119 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:07.055131 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:07.057002 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":100,"threads_started":1}
W20260812 02:38:07.057354 18837 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:07.051208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:44463 (local address 127.3.127.190:41875)
0812 02:38:07.051497 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:07.051500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:07.051567 (+ 67us) server_negotiation.cc:407] Connection header received
0812 02:38:07.051798 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:07.051802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:07.051878 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:07.051986 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:07.052584 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:07.053370 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:07.054232 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:07.054465 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:07.054542 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:07.057207 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.129:44463: BlockingRecv error: recv got EOF from 127.3.127.129:44463 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":72,"threads_started":1}
Aug 12 02:38:07 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:07 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:07.268671 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:07.262842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.130:41827)
0812 02:38:07.263130 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:07.263146 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:07.263266 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:07.263532 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:07.263535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:07.263732 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:07.263838 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:07.263844 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:07.264643 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:07.264646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:07.265296 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:07.265308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:07.265836 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:07.265839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:07.265919 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:07.266434 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:07.266539 (+ 105us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:07.268520 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
W20260812 02:38:07.268795 18837 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:07.263030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:41827 (local address 127.3.127.190:41875)
0812 02:38:07.263194 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:07.263199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:07.263214 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:38:07.263364 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:07.263367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:07.263412 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:07.263480 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:07.263978 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:07.264527 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:07.265455 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:07.265706 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:07.265790 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:07.268686 (+ 2896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.130:41827: BlockingRecv error: recv got EOF from 127.3.127.130:41827 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 12 02:38:07 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:07 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:07.465687 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:07.459628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.131:46511)
0812 02:38:07.459926 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:07.459942 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:07.460060 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:07.460465 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:07.460468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:07.460672 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:07.460783 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:07.460789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:07.461719 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:07.461723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:07.462368 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:07.462380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:07.462855 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:07.462858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:07.462923 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:07.463521 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:07.463538 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:07.465539 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":84,"threads_started":1}
W20260812 02:38:07.465821 18837 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:07.459772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:46511 (local address 127.3.127.190:41875)
0812 02:38:07.459952 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:07.459955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:07.460018 (+ 63us) server_negotiation.cc:407] Connection header received
0812 02:38:07.460236 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:07.460239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:07.460301 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:07.460384 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:07.460915 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:07.461575 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:07.462503 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:07.462752 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:07.462842 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:07.465690 (+ 2848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.131:46511: BlockingRecv error: recv got EOF from 127.3.127.131:46511 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 12 02:38:08 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:08 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:08.063962 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:08.058040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.129:58315)
0812 02:38:08.058346 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:08.058361 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:08.058462 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:08.058790 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:08.058793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:08.058990 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:08.059102 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:08.059108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:08.059954 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:08.059957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:08.060590 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:08.060601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:08.061106 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:08.061110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:08.061256 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:08.061805 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:08.061818 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:08.063801 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":96,"threads_started":1}
W20260812 02:38:08.064108 18842 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:08.058182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:58315 (local address 127.3.127.190:41875)
0812 02:38:08.058497 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:08.058501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:08.058514 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:38:08.058556 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:08.058558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:08.058604 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:08.058716 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:08.059237 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:08.059816 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:08.060721 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:08.060966 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:08.061050 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:08.063982 (+ 2932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.129:58315: BlockingRecv error: recv got EOF from 127.3.127.129:58315 (error 108)
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":65,"threads_started":1}
Aug 12 02:38:08 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:08 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:08.275271 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:08.269632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.130:44853)
0812 02:38:08.269949 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:08.269964 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:08.270095 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:08.270335 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:08.270338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:08.270530 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:08.270639 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:08.270644 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:08.271377 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:08.271379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:08.271986 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:08.271995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:08.272380 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:08.272383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:08.272446 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:08.272978 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:08.272992 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:08.275131 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":105,"threads_started":1}
W20260812 02:38:08.275420 18842 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:08.269752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:44853 (local address 127.3.127.190:41875)
0812 02:38:08.269940 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:08.269944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:08.270039 (+ 95us) server_negotiation.cc:407] Connection header received
0812 02:38:08.270179 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:08.270182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:08.270233 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:08.270304 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:08.270748 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:08.271281 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:08.272079 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:08.272295 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:08.272362 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:08.275294 (+ 2932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.130:44853: BlockingRecv error: recv got EOF from 127.3.127.130:44853 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 12 02:38:08 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:08 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.131@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:08.472337 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:08.466743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.131:55331)
0812 02:38:08.467098 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:08.467111 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:08.467245 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:08.467524 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:08.467527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:08.467773 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:08.467908 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:08.467917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:08.468741 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:08.468745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:08.469357 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:08.469367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:08.469814 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:08.469818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:08.469902 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:08.470398 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:08.470411 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:08.472195 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":98,"threads_started":1}
W20260812 02:38:08.472452 18842 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:08.466813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:55331 (local address 127.3.127.190:41875)
0812 02:38:08.466939 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:08.466943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:08.467215 (+ 272us) server_negotiation.cc:407] Connection header received
0812 02:38:08.467339 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:08.467341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:08.467398 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:08.467475 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:08.468037 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:08.468625 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:08.469474 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:08.469709 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:08.469799 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:08.472345 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.131:55331: BlockingRecv error: recv got EOF from 127.3.127.131:55331 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 12 02:38:09 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:09 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.129@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:09.071113 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:09.065050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.129:51179)
0812 02:38:09.065361 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:09.065378 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:09.065469 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:09.065693 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:09.065695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:09.065921 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:09.066069 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:09.066076 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:09.066975 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:09.066978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:09.067649 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:09.067661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:09.068247 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:09.068250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:09.068331 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:09.068948 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:09.068962 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:09.070967 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":121,"threads_started":1}
W20260812 02:38:09.071349 18848 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:09.065171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:51179 (local address 127.3.127.190:41875)
0812 02:38:09.065465 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:09.065469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:09.065483 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:38:09.065529 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:09.065531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:09.065574 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:09.065667 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:09.066196 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:09.066824 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:09.067781 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:09.068026 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:09.068115 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:09.071192 (+ 3077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.129:51179: BlockingRecv error: recv got EOF from 127.3.127.129:51179 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":110,"threads_started":1}
Aug 12 02:38:09 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:38:09 dist-test-slave-rmhg krb5kdc[17043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.127.130@KRBTEST.COM for oryx/127.3.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:38:09.282419 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:09.276324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41875 (local address 127.3.127.130:35033)
0812 02:38:09.276619 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:09.276637 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:09.276737 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:09.277055 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:09.277059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:09.277277 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:09.277386 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:09.277392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:09.278214 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:09.278216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:09.278797 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:09.278805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:09.279302 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:09.279305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:09.279377 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:09.279881 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:09.279893 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:09.282256 (+ 2363us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.127.190:41875: Server oryx/127.3.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":76,"threads_started":1}
W20260812 02:38:09.282553 18848 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:09.276491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:35033 (local address 127.3.127.190:41875)
0812 02:38:09.276705 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:09.276710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:09.276727 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:38:09.276815 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:09.276818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:09.276870 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:09.276941 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:09.277553 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:09.278121 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:09.278932 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:09.279184 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:09.279273 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:09.282430 (+ 3157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.127.130:35033: BlockingRecv error: recv got EOF from 127.3.127.130:35033 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 02:38:09.366582 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 18046
I20260812 02:38:09.383627 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 18287
I20260812 02:38:09.409294 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 18514
I20260812 02:38:09.429029 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 17933
2026-08-12T02:38:09Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18367 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T02:38:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:38:09Z Disabled control of system clock
I20260812 02:38:09.469473 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:37179
--webserver_interface=127.3.127.190
--webserver_port=0
--builtin_ntp_servers=127.3.127.148:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:37179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 02:38:09.587589 18858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:38:09.587873 18858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:38:09.587908 18858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:38:09.591979 18858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:38:09.592053 18858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:38:09.592068 18858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:38:09.592090 18858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:38:09.592104 18858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:38:09.597131 18858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:37179
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:37179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:38:09.598379 18858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:38:09.599879 18858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:38:09.609055 18858 server_base.cc:1034] running on GCE node
W20260812 02:38:09.608980 18863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:38:09.609026 18868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:38:09.608939 18864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:38:09.609906 18858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:38:09.611095 18858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:38:09.612267 18858 hybrid_clock.cc:648] HybridClock initialized: now 1786502289612245 us; error 34 us; skew 500 ppm
I20260812 02:38:09.614209 18858 webserver.cc:460] Webserver started at http://127.3.127.190:34565/ using document root <none> and password file <none>
I20260812 02:38:09.614791 18858 fs_manager.cc:359] Metadata directory not provided
I20260812 02:38:09.614861 18858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:38:09.615085 18858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:38:09.617199 18858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "6d237493c48c4628b4fc0c5399855b18"
format_stamp: "Formatted at 2026-08-12 02:38:09 on dist-test-slave-rmhg"
server_key: "d3a708f437e160127fbd7c2603d01e02"
server_key_iv: "6874d5db3dec17f15019b9a524407155"
server_key_version: "encryptionkey@0"
I20260812 02:38:09.617736 18858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "6d237493c48c4628b4fc0c5399855b18"
format_stamp: "Formatted at 2026-08-12 02:38:09 on dist-test-slave-rmhg"
server_key: "d3a708f437e160127fbd7c2603d01e02"
server_key_iv: "6874d5db3dec17f15019b9a524407155"
server_key_version: "encryptionkey@0"
I20260812 02:38:09.621450 18858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260812 02:38:09.623813 18876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:09.625005 18858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 02:38:09.625121 18858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "6d237493c48c4628b4fc0c5399855b18"
format_stamp: "Formatted at 2026-08-12 02:38:09 on dist-test-slave-rmhg"
server_key: "d3a708f437e160127fbd7c2603d01e02"
server_key_iv: "6874d5db3dec17f15019b9a524407155"
server_key_version: "encryptionkey@0"
I20260812 02:38:09.625260 18858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:38:09.636685 18858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:38:09.637497 18858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:38:09.637679 18858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:38:09.644639 18858 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:37179
I20260812 02:38:09.644654 18969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:37179 every 8 connection(s)
I20260812 02:38:09.645727 18858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:38:09.646314 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 18858
I20260812 02:38:09.646443 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:38:09.646874 3582 external_mini_cluster.cc:1428] Setting key f98d22de1dcb4a385597560c29fa3428
I20260812 02:38:09.649878 18972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:38:09.655956 18972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18: Bootstrap starting.
I20260812 02:38:09.656886 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:09.648452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39250 (local address 127.3.127.190:37179)
0812 02:38:09.649283 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:09.649301 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:09.649344 (+ 43us) server_negotiation.cc:407] Connection header received
0812 02:38:09.650107 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:09.650128 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:09.650558 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:09.650895 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:09.651364 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:09.652409 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:09.653382 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:09.653888 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:09.654041 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:09.655073 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:38:09.655078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:38:09.655091 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:38:09.655492 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:38:09.655550 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:38:09.655571 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:38:09.655661 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 02:38:09.655816 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 02:38:09.656113 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":175,"threads_started":1}
I20260812 02:38:09.658512 18972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18: Neither blocks nor log segments found. Creating new log.
I20260812 02:38:09.659377 18972 log.cc:824] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18: Log is configured to *not* fsync() on all Append() calls
I20260812 02:38:09.661602 18972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18: No bootstrap required, opened a new log
I20260812 02:38:09.665455 18972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d237493c48c4628b4fc0c5399855b18" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37179 } }
I20260812 02:38:09.665704 18972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:38:09.665755 18972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d237493c48c4628b4fc0c5399855b18, State: Initialized, Role: FOLLOWER
I20260812 02:38:09.666204 18972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d237493c48c4628b4fc0c5399855b18" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37179 } }
I20260812 02:38:09.666357 18972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:38:09.666422 18972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:38:09.666531 18972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:38:09.667454 18972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d237493c48c4628b4fc0c5399855b18" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37179 } }
I20260812 02:38:09.667804 18972 leader_election.cc:302] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d237493c48c4628b4fc0c5399855b18; no voters:
I20260812 02:38:09.668113 18972 leader_election.cc:288] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:38:09.668252 18979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:38:09.668469 18979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18 [term 1 LEADER]: Becoming Leader. State: Replica: 6d237493c48c4628b4fc0c5399855b18, State: Running, Role: LEADER
I20260812 02:38:09.668851 18979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d237493c48c4628b4fc0c5399855b18" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37179 } }
I20260812 02:38:09.669409 18972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:38:09.670439 18981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d237493c48c4628b4fc0c5399855b18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d237493c48c4628b4fc0c5399855b18" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37179 } } }
I20260812 02:38:09.670574 18981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18 [sys.catalog]: This master's current role is: LEADER
I20260812 02:38:09.671204 18982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d237493c48c4628b4fc0c5399855b18. Latest consensus state: current_term: 1 leader_uuid: "6d237493c48c4628b4fc0c5399855b18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d237493c48c4628b4fc0c5399855b18" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37179 } } }
I20260812 02:38:09.671298 18992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:38:09.671368 18982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18 [sys.catalog]: This master's current role is: LEADER
I20260812 02:38:09.674178 18992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:38:09.680691 18992 catalog_manager.cc:1347] Generated new cluster ID: a5a28ada3c37432181a67194a5970657
I20260812 02:38:09.680780 18992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:38:09.701135 18992 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:38:09.702293 18992 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:38:09.710677 18992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d237493c48c4628b4fc0c5399855b18: Generated new TSK 0
I20260812 02:38:09.711499 18992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:38:09.715926 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37179
--builtin_ntp_servers=127.3.127.148:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 02:38:09.838629 19011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:38:09.838948 19011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:38:09.838989 19011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:38:09.843300 19011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:38:09.843377 19011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:38:09.843437 19011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:38:09.848596 19011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:38:09.849865 19011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:38:09.851337 19011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:38:09.861485 19022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:38:09.861570 19018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:38:09.861593 19019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:38:09.861670 19011 server_base.cc:1034] running on GCE node
I20260812 02:38:09.862295 19011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:38:09.863036 19011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:38:09.864219 19011 hybrid_clock.cc:648] HybridClock initialized: now 1786502289864202 us; error 28 us; skew 500 ppm
I20260812 02:38:09.866360 19011 webserver.cc:460] Webserver started at http://127.3.127.129:37651/ using document root <none> and password file <none>
I20260812 02:38:09.867027 19011 fs_manager.cc:359] Metadata directory not provided
I20260812 02:38:09.867107 19011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:38:09.867324 19011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:38:09.869554 19011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "ae82a9a6136247a0acf9997e603a532f"
format_stamp: "Formatted at 2026-08-12 02:38:09 on dist-test-slave-rmhg"
server_key: "08cd5a5d22b4178a13fde910e4d4341a"
server_key_iv: "122e558eb1736fe96e76e51108cf7a05"
server_key_version: "encryptionkey@0"
I20260812 02:38:09.870198 19011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "ae82a9a6136247a0acf9997e603a532f"
format_stamp: "Formatted at 2026-08-12 02:38:09 on dist-test-slave-rmhg"
server_key: "08cd5a5d22b4178a13fde910e4d4341a"
server_key_iv: "122e558eb1736fe96e76e51108cf7a05"
server_key_version: "encryptionkey@0"
I20260812 02:38:09.874207 19011 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:38:09.876924 19032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:09.878294 19011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:38:09.878413 19011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "ae82a9a6136247a0acf9997e603a532f"
format_stamp: "Formatted at 2026-08-12 02:38:09 on dist-test-slave-rmhg"
server_key: "08cd5a5d22b4178a13fde910e4d4341a"
server_key_iv: "122e558eb1736fe96e76e51108cf7a05"
server_key_version: "encryptionkey@0"
I20260812 02:38:09.878518 19011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:38:09.890619 19011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:38:09.891402 19011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:38:09.891594 19011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:38:09.892237 19011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:38:09.893311 19011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:38:09.893364 19011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:09.893422 19011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:38:09.893450 19011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:09.902529 19011 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:46781
I20260812 02:38:09.902555 19211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:46781 every 8 connection(s)
I20260812 02:38:09.903508 19011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:38:09.903688 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 19011
I20260812 02:38:09.903802 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:38:09.904191 3582 external_mini_cluster.cc:1428] Setting key 22e77077089e3da039d7c33acefe1e30
I20260812 02:38:09.906339 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37179
--builtin_ntp_servers=127.3.127.148:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 02:38:09.914938 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:09.905598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:46801 (local address 127.3.127.190:37179)
0812 02:38:09.905734 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:09.905738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:09.907142 (+ 1404us) server_negotiation.cc:407] Connection header received
0812 02:38:09.908867 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:09.908871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:09.908935 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:09.909029 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:09.910257 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:09.910821 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:09.912560 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:09.912788 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:09.912869 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:09.914349 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:38:09.914352 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:38:09.914354 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:38:09.914369 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:38:09.914392 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:38:09.914400 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:38:09.914462 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 02:38:09.914703 (+ 241us) server_negotiation.cc:299] Negotiation successful
0812 02:38:09.914799 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 02:38:09.916242 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:09.906109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:37179 (local address 127.3.127.129:46801)
0812 02:38:09.906950 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:09.907024 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:09.908633 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:09.909282 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:09.909290 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:09.909385 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:09.910093 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:09.910116 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:09.910957 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:09.910960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:09.912434 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:09.912447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:09.912955 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:09.912960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:09.913237 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:09.913951 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:38:09.913953 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:09.914031 (+ 78us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:38:09.914138 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:38:09.914484 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:38:09.914495 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:38:09.914501 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 02:38:09.914656 (+ 155us) client_negotiation.cc:240] Negotiation successful
0812 02:38:09.915046 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":128,"threads_started":1}
I20260812 02:38:09.917841 19212 heartbeater.cc:347] Connected to a master server at 127.3.127.190:37179
I20260812 02:38:09.918120 19212 heartbeater.cc:464] Registering TS with master...
I20260812 02:38:09.918740 19212 heartbeater.cc:511] Master 127.3.127.190:37179 requested a full tablet report, sending...
I20260812 02:38:09.920428 18903 ts_manager.cc:194] Registered new tserver with Master: ae82a9a6136247a0acf9997e603a532f (127.3.127.129:46781)
I20260812 02:38:09.921763 18903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.127.129:46801
W20260812 02:38:10.031703 19217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:38:10.031991 19217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:38:10.032032 19217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:38:10.036231 19217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:38:10.036299 19217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:38:10.036352 19217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:38:10.041226 19217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:38:10.042414 19217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:38:10.043893 19217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:38:10.053710 19230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:38:10.053720 19224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:38:10.053984 19217 server_base.cc:1034] running on GCE node
W20260812 02:38:10.053800 19227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:38:10.054500 19217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:38:10.055198 19217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:38:10.056371 19217 hybrid_clock.cc:648] HybridClock initialized: now 1786502290056354 us; error 33 us; skew 500 ppm
I20260812 02:38:10.058432 19217 webserver.cc:460] Webserver started at http://127.3.127.130:45095/ using document root <none> and password file <none>
I20260812 02:38:10.059028 19217 fs_manager.cc:359] Metadata directory not provided
I20260812 02:38:10.059103 19217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:38:10.059329 19217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:38:10.061595 19217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "d2b9a5f9421b4785aa68dfe37b2e36ca"
format_stamp: "Formatted at 2026-08-12 02:38:10 on dist-test-slave-rmhg"
server_key: "4f4bd214d13f29defe3d8b3b33f681f8"
server_key_iv: "748d9bdf50eea6e37136821169a64d3e"
server_key_version: "encryptionkey@0"
I20260812 02:38:10.062191 19217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "d2b9a5f9421b4785aa68dfe37b2e36ca"
format_stamp: "Formatted at 2026-08-12 02:38:10 on dist-test-slave-rmhg"
server_key: "4f4bd214d13f29defe3d8b3b33f681f8"
server_key_iv: "748d9bdf50eea6e37136821169a64d3e"
server_key_version: "encryptionkey@0"
I20260812 02:38:10.066323 19217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:38:10.069366 19236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:10.070964 19217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:38:10.071107 19217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "d2b9a5f9421b4785aa68dfe37b2e36ca"
format_stamp: "Formatted at 2026-08-12 02:38:10 on dist-test-slave-rmhg"
server_key: "4f4bd214d13f29defe3d8b3b33f681f8"
server_key_iv: "748d9bdf50eea6e37136821169a64d3e"
server_key_version: "encryptionkey@0"
I20260812 02:38:10.071239 19217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:38:10.104067 19217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:38:10.104867 19217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:38:10.105059 19217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:38:10.105710 19217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:38:10.106765 19217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:38:10.106814 19217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:10.106869 19217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:38:10.106899 19217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:10.116017 19217 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:35753
I20260812 02:38:10.116076 19410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:35753 every 8 connection(s)
I20260812 02:38:10.117049 19217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:38:10.125591 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 19217
I20260812 02:38:10.125778 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:38:10.126145 3582 external_mini_cluster.cc:1428] Setting key 6561f83efb1503f4d417a11119dcabd2
I20260812 02:38:10.127765 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:10.119134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:41245 (local address 127.3.127.190:37179)
0812 02:38:10.119288 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:10.119292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:10.120481 (+ 1189us) server_negotiation.cc:407] Connection header received
0812 02:38:10.121995 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:10.121999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:10.122058 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:10.122141 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:10.123170 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:10.123767 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:10.125346 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:10.125704 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:10.125854 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:10.127176 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:38:10.127180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:38:10.127182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:38:10.127196 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:38:10.127219 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:38:10.127227 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:38:10.127295 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 02:38:10.127538 (+ 243us) server_negotiation.cc:299] Negotiation successful
0812 02:38:10.127618 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 02:38:10.128532 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37179
--builtin_ntp_servers=127.3.127.148:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 02:38:10.128841 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:10.119581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:37179 (local address 127.3.127.130:41245)
0812 02:38:10.120318 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:10.120362 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:10.121741 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:10.122311 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:10.122319 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:10.122412 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:10.122996 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:10.123011 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:10.123907 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:10.123910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:10.125113 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:10.125128 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:10.125928 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:10.125933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:10.126126 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:10.126817 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:38:10.126818 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:10.126869 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:38:10.126955 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:38:10.127319 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:38:10.127330 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:38:10.127338 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 02:38:10.127490 (+ 152us) client_negotiation.cc:240] Negotiation successful
0812 02:38:10.127873 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":137,"threads_started":1}
I20260812 02:38:10.130136 19411 heartbeater.cc:347] Connected to a master server at 127.3.127.190:37179
I20260812 02:38:10.130474 19411 heartbeater.cc:464] Registering TS with master...
I20260812 02:38:10.131119 19411 heartbeater.cc:511] Master 127.3.127.190:37179 requested a full tablet report, sending...
I20260812 02:38:10.132443 18903 ts_manager.cc:194] Registered new tserver with Master: d2b9a5f9421b4785aa68dfe37b2e36ca (127.3.127.130:35753)
I20260812 02:38:10.133085 18903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.127.130:41245
W20260812 02:38:10.254187 19414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:38:10.254474 19414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:38:10.254518 19414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:38:10.258800 19414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:38:10.258863 19414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:38:10.258932 19414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:38:10.264004 19414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:38:10.265257 19414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:38:10.266831 19414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:38:10.277549 19427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:38:10.277602 19414 server_base.cc:1034] running on GCE node
W20260812 02:38:10.277556 19425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:38:10.277570 19430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:38:10.278296 19414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:38:10.279028 19414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:38:10.280202 19414 hybrid_clock.cc:648] HybridClock initialized: now 1786502290280178 us; error 40 us; skew 500 ppm
I20260812 02:38:10.282382 19414 webserver.cc:460] Webserver started at http://127.3.127.131:45031/ using document root <none> and password file <none>
I20260812 02:38:10.283017 19414 fs_manager.cc:359] Metadata directory not provided
I20260812 02:38:10.283090 19414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:38:10.283309 19414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:38:10.285939 19414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "d9fbcb73bfdc4ca8a6e6fe706f055b1d"
format_stamp: "Formatted at 2026-08-12 02:38:10 on dist-test-slave-rmhg"
server_key: "0909c6864033890ebba74087395e59de"
server_key_iv: "665ccea7a5f40e2208a2de82b7db36f4"
server_key_version: "encryptionkey@0"
I20260812 02:38:10.286617 19414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "d9fbcb73bfdc4ca8a6e6fe706f055b1d"
format_stamp: "Formatted at 2026-08-12 02:38:10 on dist-test-slave-rmhg"
server_key: "0909c6864033890ebba74087395e59de"
server_key_iv: "665ccea7a5f40e2208a2de82b7db36f4"
server_key_version: "encryptionkey@0"
I20260812 02:38:10.291111 19414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 02:38:10.294277 19442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:10.295964 19414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:38:10.296128 19414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "d9fbcb73bfdc4ca8a6e6fe706f055b1d"
format_stamp: "Formatted at 2026-08-12 02:38:10 on dist-test-slave-rmhg"
server_key: "0909c6864033890ebba74087395e59de"
server_key_iv: "665ccea7a5f40e2208a2de82b7db36f4"
server_key_version: "encryptionkey@0"
I20260812 02:38:10.296289 19414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:38:10.312727 19414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:38:10.313609 19414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:38:10.313807 19414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:38:10.314468 19414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:38:10.315552 19414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:38:10.315603 19414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:10.315649 19414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:38:10.315677 19414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:10.324929 19414 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:43843
I20260812 02:38:10.324957 19619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:43843 every 8 connection(s)
I20260812 02:38:10.325990 19414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:38:10.328209 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 19414
I20260812 02:38:10.328354 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:38:10.328692 3582 external_mini_cluster.cc:1428] Setting key 2323ecac6a19a324918d6aad137473f4
I20260812 02:38:10.336385 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:10.328017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:49965 (local address 127.3.127.190:37179)
0812 02:38:10.328152 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:10.328157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:10.329706 (+ 1549us) server_negotiation.cc:407] Connection header received
0812 02:38:10.331496 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:10.331499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:10.331550 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:10.331626 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:10.332560 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:10.333113 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:10.334397 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:10.334627 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:10.334711 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:10.335888 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:38:10.335891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:38:10.335894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:38:10.335910 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:38:10.335932 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:38:10.335939 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:38:10.335995 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 02:38:10.336162 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 02:38:10.336233 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 02:38:10.337373 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:10.328610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:37179 (local address 127.3.127.131:49965)
0812 02:38:10.329545 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:10.329609 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:10.331263 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:10.331774 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:10.331781 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:10.331870 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:10.332408 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:10.332422 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:10.333269 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:10.333272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:10.334268 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:10.334279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:10.334787 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:10.334790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:10.334945 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:10.335581 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:38:10.335583 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:10.335629 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:38:10.335716 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:38:10.336010 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:38:10.336020 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:38:10.336027 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 02:38:10.336155 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 02:38:10.336472 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":175,"threads_started":1}
I20260812 02:38:10.338838 19620 heartbeater.cc:347] Connected to a master server at 127.3.127.190:37179
I20260812 02:38:10.339133 19620 heartbeater.cc:464] Registering TS with master...
I20260812 02:38:10.339797 19620 heartbeater.cc:511] Master 127.3.127.190:37179 requested a full tablet report, sending...
I20260812 02:38:10.341089 18903 ts_manager.cc:194] Registered new tserver with Master: d9fbcb73bfdc4ca8a6e6fe706f055b1d (127.3.127.131:43843)
I20260812 02:38:10.341897 18903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.127.131:49965
I20260812 02:38:10.343289 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 02:38:10.346271 18976 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:10.345353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39256 (local address 127.3.127.190:37179)
0812 02:38:10.345505 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:10.345510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:10.345677 (+ 167us) server_negotiation.cc:407] Connection header received
0812 02:38:10.345752 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:10.345754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:10.345798 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:10.345866 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:10.346122 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39256: BlockingRecv error: recv got EOF from 127.0.0.1:39256 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 02:38:10.346397 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 19011
I20260812 02:38:10.384019 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 19217
I20260812 02:38:10.412273 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 19414
I20260812 02:38:10.436868 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 18858
2026-08-12T02:38:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1008 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T02:38:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:38:10Z Disabled control of system clock
I20260812 02:38:10.477846 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:39567
--webserver_interface=127.3.127.190
--webserver_port=0
--builtin_ntp_servers=127.3.127.148:46407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:39567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 02:38:10.597143 19636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:38:10.597484 19636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:38:10.597524 19636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:38:10.601660 19636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:38:10.601732 19636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:38:10.601747 19636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:38:10.601761 19636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:38:10.601775 19636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:38:10.606841 19636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:46407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:39567
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:39567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:38:10.608083 19636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:38:10.609578 19636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:38:10.617944 19646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:38:10.617983 19636 server_base.cc:1034] running on GCE node
W20260812 02:38:10.617929 19649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:38:10.617929 19643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:38:10.618714 19636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:38:10.619804 19636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:38:10.620970 19636 hybrid_clock.cc:648] HybridClock initialized: now 1786502290620953 us; error 31 us; skew 500 ppm
I20260812 02:38:10.622943 19636 webserver.cc:460] Webserver started at http://127.3.127.190:38165/ using document root <none> and password file <none>
I20260812 02:38:10.623505 19636 fs_manager.cc:359] Metadata directory not provided
I20260812 02:38:10.623577 19636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:38:10.623780 19636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:38:10.625941 19636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "f15d37b091ec4290acc6eb66deca2296"
format_stamp: "Formatted at 2026-08-12 02:38:10 on dist-test-slave-rmhg"
server_key: "c56ae64a23639d0b1f3db662fdfb9628"
server_key_iv: "ca891de778169e912f5a22730ac1845c"
server_key_version: "encryptionkey@0"
I20260812 02:38:10.626502 19636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "f15d37b091ec4290acc6eb66deca2296"
format_stamp: "Formatted at 2026-08-12 02:38:10 on dist-test-slave-rmhg"
server_key: "c56ae64a23639d0b1f3db662fdfb9628"
server_key_iv: "ca891de778169e912f5a22730ac1845c"
server_key_version: "encryptionkey@0"
I20260812 02:38:10.630275 19636 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:38:10.632812 19657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:10.634128 19636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 02:38:10.634249 19636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "f15d37b091ec4290acc6eb66deca2296"
format_stamp: "Formatted at 2026-08-12 02:38:10 on dist-test-slave-rmhg"
server_key: "c56ae64a23639d0b1f3db662fdfb9628"
server_key_iv: "ca891de778169e912f5a22730ac1845c"
server_key_version: "encryptionkey@0"
I20260812 02:38:10.634354 19636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:38:10.639107 19636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:38:10.639747 19636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:38:10.639909 19636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:38:10.647073 19636 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:39567
I20260812 02:38:10.647085 19743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:39567 every 8 connection(s)
I20260812 02:38:10.648057 19636 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:38:10.651548 19745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:38:10.654595 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 19636
I20260812 02:38:10.654747 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:38:10.655073 3582 external_mini_cluster.cc:1428] Setting key ef40cc600949b72135179c48d7d1bc02
I20260812 02:38:10.657979 19745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296: Bootstrap starting.
I20260812 02:38:10.660552 19745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296: Neither blocks nor log segments found. Creating new log.
I20260812 02:38:10.660923 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:10.656579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58692 (local address 127.3.127.190:39567)
0812 02:38:10.657271 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:10.657288 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:10.657326 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:38:10.657975 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:10.657993 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:10.658397 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:10.658703 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:10.659569 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:38:10.659574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:38:10.659586 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:38:10.659597 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:38:10.659656 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:38:10.659673 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:38:10.659781 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 02:38:10.659874 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 02:38:10.660128 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":124,"threads_started":1}
I20260812 02:38:10.661396 19745 log.cc:824] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296: Log is configured to *not* fsync() on all Append() calls
I20260812 02:38:10.663574 19745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296: No bootstrap required, opened a new log
I20260812 02:38:10.667169 19745 raft_consensus.cc:348] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15d37b091ec4290acc6eb66deca2296" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 39567 } }
I20260812 02:38:10.667424 19745 raft_consensus.cc:374] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:38:10.667464 19745 raft_consensus.cc:729] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f15d37b091ec4290acc6eb66deca2296, State: Initialized, Role: FOLLOWER
I20260812 02:38:10.667932 19745 consensus_queue.cc:260] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15d37b091ec4290acc6eb66deca2296" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 39567 } }
I20260812 02:38:10.668104 19745 raft_consensus.cc:388] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:38:10.668183 19745 raft_consensus.cc:482] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:38:10.668282 19745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:38:10.669318 19745 raft_consensus.cc:504] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15d37b091ec4290acc6eb66deca2296" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 39567 } }
I20260812 02:38:10.669669 19745 leader_election.cc:302] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f15d37b091ec4290acc6eb66deca2296; no voters:
I20260812 02:38:10.669977 19745 leader_election.cc:288] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:38:10.670092 19757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:38:10.670316 19757 raft_consensus.cc:686] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296 [term 1 LEADER]: Becoming Leader. State: Replica: f15d37b091ec4290acc6eb66deca2296, State: Running, Role: LEADER
I20260812 02:38:10.670674 19757 consensus_queue.cc:237] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15d37b091ec4290acc6eb66deca2296" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 39567 } }
I20260812 02:38:10.671221 19745 sys_catalog.cc:564] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:38:10.672362 19760 sys_catalog.cc:455] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f15d37b091ec4290acc6eb66deca2296. Latest consensus state: current_term: 1 leader_uuid: "f15d37b091ec4290acc6eb66deca2296" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15d37b091ec4290acc6eb66deca2296" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 39567 } } }
I20260812 02:38:10.672501 19760 sys_catalog.cc:458] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296 [sys.catalog]: This master's current role is: LEADER
I20260812 02:38:10.672879 19758 sys_catalog.cc:455] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f15d37b091ec4290acc6eb66deca2296" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15d37b091ec4290acc6eb66deca2296" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 39567 } } }
I20260812 02:38:10.673022 19758 sys_catalog.cc:458] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296 [sys.catalog]: This master's current role is: LEADER
I20260812 02:38:10.672935 19771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:38:10.676817 19771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:38:10.682997 19771 catalog_manager.cc:1347] Generated new cluster ID: 2d9544e2b94b418c83e8965d2386f57a
I20260812 02:38:10.683092 19771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:38:10.702157 19771 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:38:10.702603 19771 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:38:10.712968 19771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f15d37b091ec4290acc6eb66deca2296: Generated new TSK 0
I20260812 02:38:10.713816 19771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:38:10.723416 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:39567
--builtin_ntp_servers=127.3.127.148:46407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 02:38:10.845577 19782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:38:10.845893 19782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:38:10.845958 19782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:38:10.850201 19782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:38:10.850272 19782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:38:10.850354 19782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:38:10.855475 19782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:46407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:39567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:38:10.856890 19782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:38:10.858508 19782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:38:10.868914 19790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:38:10.868991 19792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:38:10.868942 19795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:38:10.869223 19782 server_base.cc:1034] running on GCE node
I20260812 02:38:10.869836 19782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:38:10.870581 19782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:38:10.871776 19782 hybrid_clock.cc:648] HybridClock initialized: now 1786502290871736 us; error 50 us; skew 500 ppm
I20260812 02:38:10.873910 19782 webserver.cc:460] Webserver started at http://127.3.127.129:36103/ using document root <none> and password file <none>
I20260812 02:38:10.874545 19782 fs_manager.cc:359] Metadata directory not provided
I20260812 02:38:10.874614 19782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:38:10.874830 19782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:38:10.877120 19782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "aacd46d5d178453981af331c925e3ee3"
format_stamp: "Formatted at 2026-08-12 02:38:10 on dist-test-slave-rmhg"
server_key: "f510591fba4a29268b66f5f1f552c935"
server_key_iv: "d475687b61b79e1095f716833574b58a"
server_key_version: "encryptionkey@0"
I20260812 02:38:10.877744 19782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "aacd46d5d178453981af331c925e3ee3"
format_stamp: "Formatted at 2026-08-12 02:38:10 on dist-test-slave-rmhg"
server_key: "f510591fba4a29268b66f5f1f552c935"
server_key_iv: "d475687b61b79e1095f716833574b58a"
server_key_version: "encryptionkey@0"
I20260812 02:38:10.881855 19782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:38:10.884546 19801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:10.885974 19782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:38:10.886102 19782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "aacd46d5d178453981af331c925e3ee3"
format_stamp: "Formatted at 2026-08-12 02:38:10 on dist-test-slave-rmhg"
server_key: "f510591fba4a29268b66f5f1f552c935"
server_key_iv: "d475687b61b79e1095f716833574b58a"
server_key_version: "encryptionkey@0"
I20260812 02:38:10.886219 19782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:38:10.890914 19782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:38:10.891719 19782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:38:10.891924 19782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:38:10.892467 19782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:38:10.893446 19782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:38:10.893493 19782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:10.893543 19782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:38:10.893563 19782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:10.902575 19782 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:40481
I20260812 02:38:10.902602 19970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:40481 every 8 connection(s)
I20260812 02:38:10.903558 19782 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:38:10.910228 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:10.905526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:54153 (local address 127.3.127.190:39567)
0812 02:38:10.905665 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:10.905669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:10.906959 (+ 1290us) server_negotiation.cc:407] Connection header received
0812 02:38:10.908357 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:10.908360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:10.908416 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:10.908496 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:10.909749 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:38:10.909753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:38:10.909756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:38:10.909772 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:38:10.909793 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:38:10.909802 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:38:10.909888 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 02:38:10.910013 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 02:38:10.910076 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 02:38:10.911171 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:10.905985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:39567 (local address 127.3.127.129:54153)
0812 02:38:10.906779 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:10.906825 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:10.908110 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:10.908705 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:10.908713 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:10.908816 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:10.909457 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:38:10.909459 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:10.909503 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:38:10.909590 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:38:10.909872 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:38:10.909881 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:38:10.909887 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 02:38:10.910062 (+ 175us) client_negotiation.cc:240] Negotiation successful
0812 02:38:10.910334 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":132,"threads_started":1}
I20260812 02:38:10.911667 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 19782
I20260812 02:38:10.911792 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:38:10.912170 3582 external_mini_cluster.cc:1428] Setting key df3a73359060030ca14cdfdbdf78e31f
I20260812 02:38:10.912600 19971 heartbeater.cc:347] Connected to a master server at 127.3.127.190:39567
I20260812 02:38:10.912845 19971 heartbeater.cc:464] Registering TS with master...
I20260812 02:38:10.913328 19971 heartbeater.cc:511] Master 127.3.127.190:39567 requested a full tablet report, sending...
I20260812 02:38:10.914211 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:39567
--builtin_ntp_servers=127.3.127.148:46407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 02:38:10.915151 19688 ts_manager.cc:194] Registered new tserver with Master: aacd46d5d178453981af331c925e3ee3 (127.3.127.129:40481)
W20260812 02:38:11.037137 19975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:38:11.037519 19975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:38:11.037565 19975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:38:11.041889 19975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:38:11.041960 19975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:38:11.042024 19975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:38:11.047109 19975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:46407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:39567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:38:11.048331 19975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:38:11.049899 19975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:38:11.060462 19984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:38:11.060427 19982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:38:11.060602 19981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:38:11.060714 19975 server_base.cc:1034] running on GCE node
I20260812 02:38:11.061244 19975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:38:11.061939 19975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:38:11.063112 19975 hybrid_clock.cc:648] HybridClock initialized: now 1786502291063089 us; error 38 us; skew 500 ppm
I20260812 02:38:11.065263 19975 webserver.cc:460] Webserver started at http://127.3.127.130:35023/ using document root <none> and password file <none>
I20260812 02:38:11.065881 19975 fs_manager.cc:359] Metadata directory not provided
I20260812 02:38:11.065958 19975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:38:11.066179 19975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:38:11.068508 19975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "de89f2aa447e4773933fca2fb911fb58"
format_stamp: "Formatted at 2026-08-12 02:38:11 on dist-test-slave-rmhg"
server_key: "e8e0cf91355aa31c5387c6d6ec4b1bc9"
server_key_iv: "53f37ac2b2d43ff0f8f7497d8c0db3cc"
server_key_version: "encryptionkey@0"
I20260812 02:38:11.069182 19975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "de89f2aa447e4773933fca2fb911fb58"
format_stamp: "Formatted at 2026-08-12 02:38:11 on dist-test-slave-rmhg"
server_key: "e8e0cf91355aa31c5387c6d6ec4b1bc9"
server_key_iv: "53f37ac2b2d43ff0f8f7497d8c0db3cc"
server_key_version: "encryptionkey@0"
I20260812 02:38:11.073690 19975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:38:11.076898 19994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:11.078446 19975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:38:11.078608 19975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "de89f2aa447e4773933fca2fb911fb58"
format_stamp: "Formatted at 2026-08-12 02:38:11 on dist-test-slave-rmhg"
server_key: "e8e0cf91355aa31c5387c6d6ec4b1bc9"
server_key_iv: "53f37ac2b2d43ff0f8f7497d8c0db3cc"
server_key_version: "encryptionkey@0"
I20260812 02:38:11.078751 19975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:38:11.083936 19975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:38:11.084710 19975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:38:11.084894 19975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:38:11.085565 19975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:38:11.086640 19975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:38:11.086704 19975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:11.086762 19975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:38:11.086786 19975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:11.096019 19975 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:33045
I20260812 02:38:11.096076 20199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:33045 every 8 connection(s)
I20260812 02:38:11.097093 19975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:38:11.102488 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 19975
I20260812 02:38:11.102622 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:38:11.102916 3582 external_mini_cluster.cc:1428] Setting key c2cae5bb1f70893679adecfcc66131e3
I20260812 02:38:11.103950 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:11.099261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:43561 (local address 127.3.127.190:39567)
0812 02:38:11.099398 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:11.099401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:11.100666 (+ 1265us) server_negotiation.cc:407] Connection header received
0812 02:38:11.102140 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:11.102143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:11.102197 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:11.102284 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:11.103488 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:38:11.103491 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:38:11.103494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:38:11.103509 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:38:11.103530 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:38:11.103539 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:38:11.103627 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 02:38:11.103760 (+ 133us) server_negotiation.cc:299] Negotiation successful
0812 02:38:11.103809 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 02:38:11.104990 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:11.099659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:39567 (local address 127.3.127.130:43561)
0812 02:38:11.100451 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:11.100507 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:11.101894 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:11.102485 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:11.102493 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:11.102601 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:11.103193 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:38:11.103194 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:11.103241 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:38:11.103322 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:38:11.103619 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:38:11.103628 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:38:11.103636 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 02:38:11.103812 (+ 176us) client_negotiation.cc:240] Negotiation successful
0812 02:38:11.104106 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":140,"threads_started":1}
I20260812 02:38:11.105028 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:39567
--builtin_ntp_servers=127.3.127.148:46407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 02:38:11.106244 20202 heartbeater.cc:347] Connected to a master server at 127.3.127.190:39567
I20260812 02:38:11.106519 20202 heartbeater.cc:464] Registering TS with master...
I20260812 02:38:11.107039 20202 heartbeater.cc:511] Master 127.3.127.190:39567 requested a full tablet report, sending...
I20260812 02:38:11.108186 19688 ts_manager.cc:194] Registered new tserver with Master: de89f2aa447e4773933fca2fb911fb58 (127.3.127.130:33045)
W20260812 02:38:11.230628 20205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:38:11.230968 20205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:38:11.231014 20205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:38:11.235368 20205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:38:11.235445 20205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:38:11.235507 20205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:38:11.240706 20205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:46407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:39567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:38:11.241997 20205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:38:11.243566 20205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:38:11.253925 20216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:38:11.253948 20213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:38:11.253948 20211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:38:11.254143 20205 server_base.cc:1034] running on GCE node
I20260812 02:38:11.254666 20205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:38:11.255370 20205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:38:11.256531 20205 hybrid_clock.cc:648] HybridClock initialized: now 1786502291256487 us; error 62 us; skew 500 ppm
I20260812 02:38:11.258605 20205 webserver.cc:460] Webserver started at http://127.3.127.131:34677/ using document root <none> and password file <none>
I20260812 02:38:11.259246 20205 fs_manager.cc:359] Metadata directory not provided
I20260812 02:38:11.259315 20205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:38:11.259521 20205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:38:11.261883 20205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "4cbb9b0116754a3a992fb0c465cbc94d"
format_stamp: "Formatted at 2026-08-12 02:38:11 on dist-test-slave-rmhg"
server_key: "4fff06259d1baf4d531466f215c56e64"
server_key_iv: "dcfde929a8a2324446bcfd2117f7031d"
server_key_version: "encryptionkey@0"
I20260812 02:38:11.262482 20205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "4cbb9b0116754a3a992fb0c465cbc94d"
format_stamp: "Formatted at 2026-08-12 02:38:11 on dist-test-slave-rmhg"
server_key: "4fff06259d1baf4d531466f215c56e64"
server_key_iv: "dcfde929a8a2324446bcfd2117f7031d"
server_key_version: "encryptionkey@0"
I20260812 02:38:11.266784 20205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260812 02:38:11.269695 20223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:11.271118 20205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:38:11.271261 20205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "4cbb9b0116754a3a992fb0c465cbc94d"
format_stamp: "Formatted at 2026-08-12 02:38:11 on dist-test-slave-rmhg"
server_key: "4fff06259d1baf4d531466f215c56e64"
server_key_iv: "dcfde929a8a2324446bcfd2117f7031d"
server_key_version: "encryptionkey@0"
I20260812 02:38:11.271391 20205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:38:11.276701 20205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:38:11.277501 20205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:38:11.277701 20205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:38:11.278304 20205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:38:11.279310 20205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:38:11.279362 20205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:11.279418 20205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:38:11.279443 20205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:11.288623 20205 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:45177
I20260812 02:38:11.288715 20404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:45177 every 8 connection(s)
I20260812 02:38:11.289778 20205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:38:11.294243 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 20205
I20260812 02:38:11.294385 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:38:11.294749 3582 external_mini_cluster.cc:1428] Setting key 65d52c0fb7318567793e4cd83fef444e
I20260812 02:38:11.301893 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:11.291916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:37009 (local address 127.3.127.190:39567)
0812 02:38:11.292085 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:11.292090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:11.293412 (+ 1322us) server_negotiation.cc:407] Connection header received
0812 02:38:11.295238 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:11.295242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:11.295300 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:11.295389 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:11.301272 (+ 5883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:38:11.301276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:38:11.301279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:38:11.301297 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:38:11.301324 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:38:11.301334 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:38:11.301437 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 02:38:11.301705 (+ 268us) server_negotiation.cc:299] Negotiation successful
0812 02:38:11.301763 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 02:38:11.302953 20408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:11.292405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:39567 (local address 127.3.127.131:37009)
0812 02:38:11.293239 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:11.293290 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:11.294853 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:11.295694 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:11.295703 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:11.295847 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:11.300790 (+ 4943us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:38:11.300793 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:11.300854 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:38:11.300968 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:38:11.301476 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:38:11.301492 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:38:11.301499 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 02:38:11.301706 (+ 207us) client_negotiation.cc:240] Negotiation successful
0812 02:38:11.301977 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":148,"threads_started":1}
I20260812 02:38:11.304608 20405 heartbeater.cc:347] Connected to a master server at 127.3.127.190:39567
I20260812 02:38:11.304908 20405 heartbeater.cc:464] Registering TS with master...
I20260812 02:38:11.305455 20405 heartbeater.cc:511] Master 127.3.127.190:39567 requested a full tablet report, sending...
I20260812 02:38:11.306707 19688 ts_manager.cc:194] Registered new tserver with Master: 4cbb9b0116754a3a992fb0c465cbc94d (127.3.127.131:45177)
I20260812 02:38:11.309710 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 02:38:11.312757 19752 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:38:11.311856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58704 (local address 127.3.127.190:39567)
0812 02:38:11.312049 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:11.312053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:11.312173 (+ 120us) server_negotiation.cc:407] Connection header received
0812 02:38:11.312264 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:11.312267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:11.312309 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:11.312381 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:11.312604 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58704: BlockingRecv error: recv got EOF from 127.0.0.1:58704 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 02:38:11.312980 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 19782
I20260812 02:38:11.326841 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 19975
I20260812 02:38:11.343147 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 20205
I20260812 02:38:11.360636 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 19636
2026-08-12T02:38:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (919 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:38:11 dist-test-slave-rmhg krb5kdc[20420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:38:11 dist-test-slave-rmhg krb5kdc[20420](info): set up 2 sockets
Aug 12 02:38:11 dist-test-slave-rmhg krb5kdc[20420](info): commencing operation
krb5kdc: starting...
W20260812 02:38:12.541322 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:38:12 dist-test-slave-rmhg krb5kdc[20420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:38:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:38:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:38:12.682801 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:33109
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:33109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:38:12.804939 20444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:38:12.805344 20444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:38:12.805424 20444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:38:12.809785 20444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:38:12.809880 20444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:38:12.809906 20444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:38:12.809931 20444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:38:12.809962 20444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:38:12.815922 20444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:33109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:33109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:38:12.817314 20444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:38:12.818543 20444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:38:12.828032 20458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:38:12.828052 20460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:38:12.828299 20444 server_base.cc:1034] running on GCE node
W20260812 02:38:12.828052 20453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:38:12.829032 20444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:38:12.830372 20444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:38:12.831562 20444 hybrid_clock.cc:648] HybridClock initialized: now 1786502292831529 us; error 49 us; skew 500 ppm
Aug 12 02:38:12 dist-test-slave-rmhg krb5kdc[20420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502292, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:38:12.834654 20444 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:38:12.835655 20444 webserver.cc:460] Webserver started at http://127.3.127.190:35663/ using document root <none> and password file <none>
I20260812 02:38:12.836227 20444 fs_manager.cc:359] Metadata directory not provided
I20260812 02:38:12.836288 20444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:38:12.836476 20444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:38:12.838683 20444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "b3cdc833159f41f485d16670c93bbce1"
format_stamp: "Formatted at 2026-08-12 02:38:12 on dist-test-slave-rmhg"
server_key: "1db5fa2dafd8da5f998b61ee07fb784e"
server_key_iv: "3ea6d22cefec1d9c993d3d2387f7acc9"
server_key_version: "encryptionkey@0"
I20260812 02:38:12.839269 20444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "b3cdc833159f41f485d16670c93bbce1"
format_stamp: "Formatted at 2026-08-12 02:38:12 on dist-test-slave-rmhg"
server_key: "1db5fa2dafd8da5f998b61ee07fb784e"
server_key_iv: "3ea6d22cefec1d9c993d3d2387f7acc9"
server_key_version: "encryptionkey@0"
I20260812 02:38:12.843454 20444 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:38:12.846155 20468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:12.847522 20444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:38:12.847680 20444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "b3cdc833159f41f485d16670c93bbce1"
format_stamp: "Formatted at 2026-08-12 02:38:12 on dist-test-slave-rmhg"
server_key: "1db5fa2dafd8da5f998b61ee07fb784e"
server_key_iv: "3ea6d22cefec1d9c993d3d2387f7acc9"
server_key_version: "encryptionkey@0"
I20260812 02:38:12.847790 20444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:38:12.858438 20444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:38:12.861485 20444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:38:12.861662 20444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:38:12.868825 20444 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:33109
I20260812 02:38:12.869201 20552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:33109 every 8 connection(s)
I20260812 02:38:12.869956 20444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:38:12.871091 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 20444
I20260812 02:38:12.871227 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:38:12.871554 3582 external_mini_cluster.cc:1428] Setting key 379fd00785f2f075b3a14bc42dd15264
I20260812 02:38:12.873703 20553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:38:12.880532 20553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1: Bootstrap starting.
Aug 12 02:38:12 dist-test-slave-rmhg krb5kdc[20420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:38:12.882956 20553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1: Neither blocks nor log segments found. Creating new log.
I20260812 02:38:12.883713 20553 log.cc:824] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1: Log is configured to *not* fsync() on all Append() calls
I20260812 02:38:12.885847 20553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1: No bootstrap required, opened a new log
I20260812 02:38:12.886214 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:12.873054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39818 (local address 127.3.127.190:33109)
0812 02:38:12.873533 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:12.873550 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:12.873589 (+ 39us) server_negotiation.cc:407] Connection header received
0812 02:38:12.874291 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:12.874311 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:12.874658 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:12.874967 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:12.875838 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:12.876918 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:12.878094 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:12.878598 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:12.878724 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:12.881557 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:38:12.881563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:38:12.881577 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:38:12.881610 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:38:12.883266 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:38:12.883707 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:38:12.883712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:38:12.883717 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:38:12.883842 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:38:12.884106 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:38:12.884109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:38:12.884111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:38:12.884555 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:38:12.884760 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0812 02:38:12.885056 (+ 296us) server_negotiation.cc:299] Negotiation successful
0812 02:38:12.885397 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":131,"threads_started":1}
I20260812 02:38:12.889761 20553 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cdc833159f41f485d16670c93bbce1" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33109 } }
I20260812 02:38:12.890018 20553 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:38:12.890096 20553 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3cdc833159f41f485d16670c93bbce1, State: Initialized, Role: FOLLOWER
I20260812 02:38:12.890559 20553 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cdc833159f41f485d16670c93bbce1" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33109 } }
I20260812 02:38:12.890725 20553 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:38:12.890798 20553 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:38:12.890908 20553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:38:12.891970 20553 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cdc833159f41f485d16670c93bbce1" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33109 } }
I20260812 02:38:12.892361 20553 leader_election.cc:302] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3cdc833159f41f485d16670c93bbce1; no voters:
I20260812 02:38:12.892738 20553 leader_election.cc:288] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:38:12.892894 20560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:38:12.893183 20560 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1 [term 1 LEADER]: Becoming Leader. State: Replica: b3cdc833159f41f485d16670c93bbce1, State: Running, Role: LEADER
I20260812 02:38:12.893503 20560 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cdc833159f41f485d16670c93bbce1" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33109 } }
I20260812 02:38:12.894237 20553 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:38:12.895498 20562 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3cdc833159f41f485d16670c93bbce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cdc833159f41f485d16670c93bbce1" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33109 } } }
I20260812 02:38:12.895651 20562 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1 [sys.catalog]: This master's current role is: LEADER
I20260812 02:38:12.895915 20563 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3cdc833159f41f485d16670c93bbce1. Latest consensus state: current_term: 1 leader_uuid: "b3cdc833159f41f485d16670c93bbce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cdc833159f41f485d16670c93bbce1" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33109 } } }
I20260812 02:38:12.896019 20563 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1 [sys.catalog]: This master's current role is: LEADER
I20260812 02:38:12.896171 20577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:38:12.899593 20577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:38:12.906250 20577 catalog_manager.cc:1347] Generated new cluster ID: 34a9b7261cad4c649d1aa1674f48981a
I20260812 02:38:12.906353 20577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:38:12.928865 20577 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:38:12.930042 20577 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:38:12.934468 20577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1: Generated new TSK 0
I20260812 02:38:12.935215 20577 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
I20260812 02:38:12.972539 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33109
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:38:13.095541 20593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:38:13.095920 20593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:38:13.096002 20593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:38:13.100405 20593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:38:13.100485 20593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:38:13.100577 20593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:38:13.106366 20593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:38:13.107599 20593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:38:13.108736 20593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:38:13.118759 20602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:38:13.118848 20606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:38:13.118788 20601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:38:13.119050 20593 server_base.cc:1034] running on GCE node
I20260812 02:38:13.119654 20593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:38:13.120458 20593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:38:13.121632 20593 hybrid_clock.cc:648] HybridClock initialized: now 1786502293121603 us; error 46 us; skew 500 ppm
Aug 12 02:38:13 dist-test-slave-rmhg krb5kdc[20420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502293, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:38:13.124713 20593 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:38:13.125823 20593 webserver.cc:460] Webserver started at http://127.3.127.129:36879/ using document root <none> and password file <none>
I20260812 02:38:13.126448 20593 fs_manager.cc:359] Metadata directory not provided
I20260812 02:38:13.126540 20593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:38:13.126813 20593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:38:13.129086 20593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "75147e8e2c864cbea9c1896b909aad05"
format_stamp: "Formatted at 2026-08-12 02:38:13 on dist-test-slave-rmhg"
server_key: "a0433d358ae90ffca70a7333357d4ac4"
server_key_iv: "10aaa78d61097d230484945f9c66c18f"
server_key_version: "encryptionkey@0"
I20260812 02:38:13.129765 20593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "75147e8e2c864cbea9c1896b909aad05"
format_stamp: "Formatted at 2026-08-12 02:38:13 on dist-test-slave-rmhg"
server_key: "a0433d358ae90ffca70a7333357d4ac4"
server_key_iv: "10aaa78d61097d230484945f9c66c18f"
server_key_version: "encryptionkey@0"
I20260812 02:38:13.133996 20593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:38:13.136797 20614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:13.138273 20593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:38:13.138389 20593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "75147e8e2c864cbea9c1896b909aad05"
format_stamp: "Formatted at 2026-08-12 02:38:13 on dist-test-slave-rmhg"
server_key: "a0433d358ae90ffca70a7333357d4ac4"
server_key_iv: "10aaa78d61097d230484945f9c66c18f"
server_key_version: "encryptionkey@0"
I20260812 02:38:13.138515 20593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:38:13.157866 20593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:38:13.161612 20593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:38:13.161826 20593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:38:13.162482 20593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:38:13.163391 20593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:38:13.163441 20593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:13.163484 20593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:38:13.163513 20593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:13.173002 20593 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:35285
I20260812 02:38:13.173040 20780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:35285 every 8 connection(s)
I20260812 02:38:13.174029 20593 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:38:13.181429 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 20593
I20260812 02:38:13.181560 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:38:13.181844 3582 external_mini_cluster.cc:1428] Setting key 8a69171fa0c325d68d2059191f5760ee
Aug 12 02:38:13 dist-test-slave-rmhg krb5kdc[20420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502293, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:38:13.189074 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:13.176039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:49511 (local address 127.3.127.190:33109)
0812 02:38:13.176249 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:13.176253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:13.177133 (+ 880us) server_negotiation.cc:407] Connection header received
0812 02:38:13.178605 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:13.178608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:13.178670 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:13.178749 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:13.180068 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.180624 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.181956 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.182209 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.182324 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:13.185075 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:38:13.185079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:38:13.185082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:38:13.185122 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:38:13.186615 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:38:13.187235 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:38:13.187238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:38:13.187239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:38:13.187334 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:38:13.187771 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:38:13.187774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:38:13.187775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:38:13.187984 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:38:13.188066 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 02:38:13.188778 (+ 712us) server_negotiation.cc:299] Negotiation successful
0812 02:38:13.188928 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260812 02:38:13.190447 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:13.176441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:33109 (local address 127.3.127.129:49511)
0812 02:38:13.176957 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:13.177006 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:13.178373 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:13.178952 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:13.178960 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:13.179353 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:13.179895 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:13.179912 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.180762 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.180765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:13.181826 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:13.181839 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.182400 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.182404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:13.182634 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:13.183307 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:13.183326 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:13.184870 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:38:13.186740 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:38:13.186747 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:38:13.186766 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:38:13.187125 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:38:13.187425 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:38:13.187428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:38:13.187430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:38:13.187677 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:38:13.188088 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:38:13.188096 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:38:13.188472 (+ 376us) client_negotiation.cc:769] Sending connection context
0812 02:38:13.188746 (+ 274us) client_negotiation.cc:240] Negotiation successful
0812 02:38:13.189194 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":147,"threads_started":1}
I20260812 02:38:13.191962 20781 heartbeater.cc:347] Connected to a master server at 127.3.127.190:33109
I20260812 02:38:13.192268 20781 heartbeater.cc:464] Registering TS with master...
I20260812 02:38:13.192994 20781 heartbeater.cc:511] Master 127.3.127.190:33109 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.127.130@KRBTEST.COM; defaulting to no policy
I20260812 02:38:13.194912 20493 ts_manager.cc:194] Registered new tserver with Master: 75147e8e2c864cbea9c1896b909aad05 (127.3.127.129:35285)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.130@KRBTEST.COM" created.
I20260812 02:38:13.196319 20493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:49511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
I20260812 02:38:13.212950 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33109
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:38:13.335628 20789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:38:13.335947 20789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:38:13.336012 20789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:38:13.340237 20789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:38:13.340332 20789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:38:13.340417 20789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:38:13.346485 20789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:38:13.347728 20789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:38:13.348852 20789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:38:13.359767 20798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:38:13.359764 20797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:38:13.359869 20789 server_base.cc:1034] running on GCE node
W20260812 02:38:13.359762 20800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:38:13.360554 20789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:38:13.361356 20789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:38:13.362562 20789 hybrid_clock.cc:648] HybridClock initialized: now 1786502293362541 us; error 32 us; skew 500 ppm
Aug 12 02:38:13 dist-test-slave-rmhg krb5kdc[20420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502293, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:38:13.365721 20789 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:38:13.366758 20789 webserver.cc:460] Webserver started at http://127.3.127.130:46005/ using document root <none> and password file <none>
I20260812 02:38:13.367349 20789 fs_manager.cc:359] Metadata directory not provided
I20260812 02:38:13.367420 20789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:38:13.367625 20789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:38:13.369889 20789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "229332a1c0b84d9bb593579ac73af6a8"
format_stamp: "Formatted at 2026-08-12 02:38:13 on dist-test-slave-rmhg"
server_key: "a96030acd28a02f8adce4ed1e16f6a6e"
server_key_iv: "92d7695964cb111d087a6039dbd7f6db"
server_key_version: "encryptionkey@0"
I20260812 02:38:13.370467 20789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "229332a1c0b84d9bb593579ac73af6a8"
format_stamp: "Formatted at 2026-08-12 02:38:13 on dist-test-slave-rmhg"
server_key: "a96030acd28a02f8adce4ed1e16f6a6e"
server_key_iv: "92d7695964cb111d087a6039dbd7f6db"
server_key_version: "encryptionkey@0"
I20260812 02:38:13.374634 20789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:38:13.377468 20813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:13.378865 20789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 02:38:13.379007 20789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "229332a1c0b84d9bb593579ac73af6a8"
format_stamp: "Formatted at 2026-08-12 02:38:13 on dist-test-slave-rmhg"
server_key: "a96030acd28a02f8adce4ed1e16f6a6e"
server_key_iv: "92d7695964cb111d087a6039dbd7f6db"
server_key_version: "encryptionkey@0"
I20260812 02:38:13.379139 20789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:38:13.390231 20789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:38:13.393707 20789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:38:13.393903 20789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:38:13.394572 20789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:38:13.395556 20789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:38:13.395619 20789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:13.395676 20789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:38:13.395709 20789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:13.404857 20789 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:36867
I20260812 02:38:13.404886 20995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:36867 every 8 connection(s)
I20260812 02:38:13.405851 20789 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:38:13.411577 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 20789
I20260812 02:38:13.411710 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:38:13.412016 3582 external_mini_cluster.cc:1428] Setting key 834a1a86f8a028d287e464fbcb454044
Aug 12 02:38:13 dist-test-slave-rmhg krb5kdc[20420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502293, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:38:13.420825 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:13.407759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:46053 (local address 127.3.127.190:33109)
0812 02:38:13.407895 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:13.407898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:13.408769 (+ 871us) server_negotiation.cc:407] Connection header received
0812 02:38:13.410189 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:13.410193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:13.410255 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:13.410325 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:13.411793 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.412419 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.413933 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.414228 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.414337 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:13.417194 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:38:13.417199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:38:13.417202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:38:13.417231 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:38:13.418630 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:38:13.419184 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:38:13.419187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:38:13.419189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:38:13.419272 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:38:13.419613 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:38:13.419616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:38:13.419618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:38:13.419811 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:38:13.419909 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 02:38:13.420463 (+ 554us) server_negotiation.cc:299] Negotiation successful
0812 02:38:13.420651 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 02:38:13.421774 21000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:13.408117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:33109 (local address 127.3.127.130:46053)
0812 02:38:13.408612 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:13.408658 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:13.409967 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:13.410525 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:13.410532 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:13.410913 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:13.411549 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:13.411572 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.412559 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.412561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:13.413803 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:13.413814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.414366 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.414369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:13.414519 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:13.415088 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:13.415106 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:13.416977 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:38:13.418745 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:38:13.418752 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:38:13.418770 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:38:13.419062 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:38:13.419359 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:38:13.419362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:38:13.419364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:38:13.419510 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:38:13.419927 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:38:13.419935 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:38:13.420221 (+ 286us) client_negotiation.cc:769] Sending connection context
0812 02:38:13.420405 (+ 184us) client_negotiation.cc:240] Negotiation successful
0812 02:38:13.420783 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.3.127.131@KRBTEST.COM; defaulting to no policy
I20260812 02:38:13.423041 20996 heartbeater.cc:347] Connected to a master server at 127.3.127.190:33109
I20260812 02:38:13.423312 20996 heartbeater.cc:464] Registering TS with master...
I20260812 02:38:13.423851 20996 heartbeater.cc:511] Master 127.3.127.190:33109 requested a full tablet report, sending...
I20260812 02:38:13.424981 20493 ts_manager.cc:194] Registered new tserver with Master: 229332a1c0b84d9bb593579ac73af6a8 (127.3.127.130:36867)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.131@KRBTEST.COM" created.
I20260812 02:38:13.425678 20493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:46053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
I20260812 02:38:13.444401 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33109
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:38:13.563684 21003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:38:13.563967 21003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:38:13.564011 21003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:38:13.568154 21003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:38:13.568236 21003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:38:13.568297 21003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:38:13.574076 21003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:33109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:38:13.575284 21003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:38:13.576375 21003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:38:13.586906 21009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:38:13.586949 21003 server_base.cc:1034] running on GCE node
W20260812 02:38:13.586906 21011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:38:13.586905 21015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:38:13.587690 21003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:38:13.588404 21003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:38:13.589583 21003 hybrid_clock.cc:648] HybridClock initialized: now 1786502293589559 us; error 40 us; skew 500 ppm
Aug 12 02:38:13 dist-test-slave-rmhg krb5kdc[20420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502293, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:38:13.592595 21003 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:38:13.593691 21003 webserver.cc:460] Webserver started at http://127.3.127.131:46807/ using document root <none> and password file <none>
I20260812 02:38:13.594352 21003 fs_manager.cc:359] Metadata directory not provided
I20260812 02:38:13.594429 21003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:38:13.594647 21003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:38:13.597129 21003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "69e83fd01a074b0e94a3af94a3c0708b"
format_stamp: "Formatted at 2026-08-12 02:38:13 on dist-test-slave-rmhg"
server_key: "246dbb821a9cff8a028e4beb904d4aca"
server_key_iv: "d653fa1bb577e93212be6244ce24aa7e"
server_key_version: "encryptionkey@0"
I20260812 02:38:13.597798 21003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "69e83fd01a074b0e94a3af94a3c0708b"
format_stamp: "Formatted at 2026-08-12 02:38:13 on dist-test-slave-rmhg"
server_key: "246dbb821a9cff8a028e4beb904d4aca"
server_key_iv: "d653fa1bb577e93212be6244ce24aa7e"
server_key_version: "encryptionkey@0"
I20260812 02:38:13.602037 21003 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:38:13.604857 21026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:13.606401 21003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:38:13.606556 21003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "69e83fd01a074b0e94a3af94a3c0708b"
format_stamp: "Formatted at 2026-08-12 02:38:13 on dist-test-slave-rmhg"
server_key: "246dbb821a9cff8a028e4beb904d4aca"
server_key_iv: "d653fa1bb577e93212be6244ce24aa7e"
server_key_version: "encryptionkey@0"
I20260812 02:38:13.606686 21003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:38:13.619194 21003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:38:13.622757 21003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:38:13.623073 21003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:38:13.623769 21003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:38:13.624704 21003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:38:13.624755 21003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:13.624812 21003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:38:13.624841 21003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:38:13.633976 21003 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:37853
I20260812 02:38:13.634002 21192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:37853 every 8 connection(s)
I20260812 02:38:13.635094 21003 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:38:13.643023 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 21003
I20260812 02:38:13.643167 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:38:13.643451 3582 external_mini_cluster.cc:1428] Setting key 0e4791a830b6d5a028a461c1ba6760e0
Aug 12 02:38:13 dist-test-slave-rmhg krb5kdc[20420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502293, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:38:13.649825 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:13.637079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:44877 (local address 127.3.127.190:33109)
0812 02:38:13.637241 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:13.637245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:13.638138 (+ 893us) server_negotiation.cc:407] Connection header received
0812 02:38:13.639613 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:13.639617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:13.639680 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:13.639754 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:13.641055 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.641651 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.642957 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.643358 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.643468 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:13.646304 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:38:13.646308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:38:13.646310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:38:13.646339 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:38:13.647716 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:38:13.648282 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:38:13.648284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:38:13.648286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:38:13.648368 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:38:13.648687 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:38:13.648689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:38:13.648691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:38:13.648878 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:38:13.648948 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 02:38:13.649571 (+ 623us) server_negotiation.cc:299] Negotiation successful
0812 02:38:13.649696 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 02:38:13.650712 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:13.637508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:33109 (local address 127.3.127.131:44877)
0812 02:38:13.637974 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:13.638020 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:13.639380 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:13.639955 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:13.639963 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:13.640369 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:38:13.640896 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:13.640912 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.641782 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.641785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:13.642808 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:13.642820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.643548 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.643552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:13.643735 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:13.644406 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:38:13.644424 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:38:13.646124 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:38:13.647845 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:38:13.647851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:38:13.647867 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:38:13.648188 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:38:13.648443 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:38:13.648445 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:38:13.648446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:38:13.648604 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:38:13.648950 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:38:13.648955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:38:13.649314 (+ 359us) client_negotiation.cc:769] Sending connection context
0812 02:38:13.649501 (+ 187us) client_negotiation.cc:240] Negotiation successful
0812 02:38:13.649812 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260812 02:38:13.652040 21193 heartbeater.cc:347] Connected to a master server at 127.3.127.190:33109
I20260812 02:38:13.652282 21193 heartbeater.cc:464] Registering TS with master...
I20260812 02:38:13.652808 21193 heartbeater.cc:511] Master 127.3.127.190:33109 requested a full tablet report, sending...
I20260812 02:38:13.653916 20493 ts_manager.cc:194] Registered new tserver with Master: 69e83fd01a074b0e94a3af94a3c0708b (127.3.127.131:37853)
I20260812 02:38:13.654485 20493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:44877
I20260812 02:38:13.657895 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:38:13.667275 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:13.659991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39832 (local address 127.3.127.190:33109)
0812 02:38:13.660161 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:13.660165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:13.660267 (+ 102us) server_negotiation.cc:407] Connection header received
0812 02:38:13.660355 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:13.660357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:13.660402 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:13.660469 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:38:13.661205 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.661742 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.662761 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.662984 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.663068 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:13.664522 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:38:13.664526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:38:13.664528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:38:13.664554 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:38:13.665820 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:38:13.666221 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:38:13.666223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:38:13.666225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:38:13.666301 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:38:13.666493 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:38:13.666495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:38:13.666496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:38:13.666725 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:38:13.666826 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 02:38:13.667008 (+ 182us) server_negotiation.cc:299] Negotiation successful
0812 02:38:13.667121 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 02:38:13.670517 20493 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:38:13.673794 20493 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:38:13.688979 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:13.682942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:35285 (local address 127.0.0.1:49978)
0812 02:38:13.684074 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:13.684143 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:13.684281 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:13.685020 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:13.685026 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:13.685060 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:38:13.685333 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:13.685340 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.686402 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.686406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:13.687560 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:13.687570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.688423 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.688426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:13.688542 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:13.688587 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 02:38:13.688698 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 02:38:13.688816 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"mutex_wait_us":91,"spinlock_wait_cycles":32768,"thread_start_us":114,"threads_started":1}
I20260812 02:38:13.688993 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:13.683378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49978 (local address 127.3.127.129:35285)
0812 02:38:13.683710 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:13.683726 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:13.684346 (+ 620us) server_negotiation.cc:407] Connection header received
0812 02:38:13.684437 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:13.684443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:13.684682 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:13.684828 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:38:13.685472 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.686268 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.687682 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.688312 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.688424 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:13.688581 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 02:38:13.688740 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 02:38:13.688840 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":91,"threads_started":1}
I20260812 02:38:13.689934 21207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:13.682818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:37853 (local address 127.0.0.1:52056)
0812 02:38:13.684069 (+ 1251us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:13.684126 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:13.684282 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:13.685020 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:13.685026 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:13.685058 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:38:13.685316 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:13.685324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.686850 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.686854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:13.688157 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:13.688168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.689444 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.689448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:13.689553 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:13.689578 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 02:38:13.689670 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 02:38:13.689801 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"mutex_wait_us":104,"spinlock_wait_cycles":22784,"thread_start_us":87,"threads_started":1}
I20260812 02:38:13.689949 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:13.682957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52056 (local address 127.3.127.131:37853)
0812 02:38:13.683794 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:13.683806 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:13.684420 (+ 614us) server_negotiation.cc:407] Connection header received
0812 02:38:13.684510 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:13.684514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:13.684673 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:13.684828 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:38:13.685444 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.686688 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.688289 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.689107 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.689239 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:13.689375 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 02:38:13.689706 (+ 331us) server_negotiation.cc:299] Negotiation successful
0812 02:38:13.689809 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":171,"threads_started":1}
I20260812 02:38:13.689998 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:13.683747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57624 (local address 127.3.127.130:36867)
0812 02:38:13.684244 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:13.684256 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:13.684649 (+ 393us) server_negotiation.cc:407] Connection header received
0812 02:38:13.684842 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:13.684846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:13.684990 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:13.685116 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:38:13.685460 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.686703 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.688351 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.689207 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.689299 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:13.689437 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 02:38:13.689791 (+ 354us) server_negotiation.cc:299] Negotiation successful
0812 02:38:13.689874 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":149,"threads_started":1}
I20260812 02:38:13.690266 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:13.683575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:36867 (local address 127.0.0.1:57624)
0812 02:38:13.684517 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:13.684530 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:13.684711 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:13.685107 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:13.685110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:13.685177 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:38:13.685337 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:13.685344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.686850 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.686854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:13.688261 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:13.688269 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.689499 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.689503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:13.689577 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:13.689596 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 02:38:13.689666 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 02:38:13.690147 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":585,"spinlock_wait_cycles":768,"thread_start_us":91,"threads_started":1}
I20260812 02:38:13.692731 20691 tablet_service.cc:1467] Processing CreateTablet for tablet 233c5ed4b52649a58619e5ab8c78b1d6 (DEFAULT_TABLE table=test-table [id=2a3548122bd84bb4987a01931e6d62e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:38:13.693251 21100 tablet_service.cc:1467] Processing CreateTablet for tablet 233c5ed4b52649a58619e5ab8c78b1d6 (DEFAULT_TABLE table=test-table [id=2a3548122bd84bb4987a01931e6d62e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:38:13.693996 20691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233c5ed4b52649a58619e5ab8c78b1d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:38:13.694355 21100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233c5ed4b52649a58619e5ab8c78b1d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:38:13.695007 20890 tablet_service.cc:1467] Processing CreateTablet for tablet 233c5ed4b52649a58619e5ab8c78b1d6 (DEFAULT_TABLE table=test-table [id=2a3548122bd84bb4987a01931e6d62e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:38:13.695915 20890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233c5ed4b52649a58619e5ab8c78b1d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:38:13.701745 21219 tablet_bootstrap.cc:492] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8: Bootstrap starting.
I20260812 02:38:13.703073 21220 tablet_bootstrap.cc:492] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05: Bootstrap starting.
I20260812 02:38:13.704834 21219 tablet_bootstrap.cc:654] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8: Neither blocks nor log segments found. Creating new log.
I20260812 02:38:13.705459 21220 tablet_bootstrap.cc:654] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05: Neither blocks nor log segments found. Creating new log.
I20260812 02:38:13.705503 21223 tablet_bootstrap.cc:492] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b: Bootstrap starting.
I20260812 02:38:13.705859 21219 log.cc:824] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8: Log is configured to *not* fsync() on all Append() calls
I20260812 02:38:13.706460 21220 log.cc:824] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05: Log is configured to *not* fsync() on all Append() calls
I20260812 02:38:13.707724 21223 tablet_bootstrap.cc:654] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b: Neither blocks nor log segments found. Creating new log.
I20260812 02:38:13.708228 21219 tablet_bootstrap.cc:492] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8: No bootstrap required, opened a new log
I20260812 02:38:13.708444 21219 ts_tablet_manager.cc:1397] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260812 02:38:13.708966 21223 log.cc:824] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b: Log is configured to *not* fsync() on all Append() calls
I20260812 02:38:13.709887 21220 tablet_bootstrap.cc:492] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05: No bootstrap required, opened a new log
I20260812 02:38:13.710108 21220 ts_tablet_manager.cc:1397] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 02:38:13.710804 21223 tablet_bootstrap.cc:492] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b: No bootstrap required, opened a new log
I20260812 02:38:13.710989 21223 ts_tablet_manager.cc:1397] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 02:38:13.712819 21219 raft_consensus.cc:348] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e83fd01a074b0e94a3af94a3c0708b" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37853 } } peers { permanent_uuid: "75147e8e2c864cbea9c1896b909aad05" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 35285 } } peers { permanent_uuid: "229332a1c0b84d9bb593579ac73af6a8" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36867 } }
I20260812 02:38:13.713140 21219 raft_consensus.cc:374] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:38:13.713230 21219 raft_consensus.cc:729] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229332a1c0b84d9bb593579ac73af6a8, State: Initialized, Role: FOLLOWER
I20260812 02:38:13.713765 21219 consensus_queue.cc:260] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e83fd01a074b0e94a3af94a3c0708b" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37853 } } peers { permanent_uuid: "75147e8e2c864cbea9c1896b909aad05" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 35285 } } peers { permanent_uuid: "229332a1c0b84d9bb593579ac73af6a8" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36867 } }
I20260812 02:38:13.714357 21223 raft_consensus.cc:348] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e83fd01a074b0e94a3af94a3c0708b" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37853 } } peers { permanent_uuid: "75147e8e2c864cbea9c1896b909aad05" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 35285 } } peers { permanent_uuid: "229332a1c0b84d9bb593579ac73af6a8" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36867 } }
I20260812 02:38:13.714637 21223 raft_consensus.cc:374] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:38:13.714692 21223 raft_consensus.cc:729] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e83fd01a074b0e94a3af94a3c0708b, State: Initialized, Role: FOLLOWER
I20260812 02:38:13.714665 21220 raft_consensus.cc:348] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e83fd01a074b0e94a3af94a3c0708b" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37853 } } peers { permanent_uuid: "75147e8e2c864cbea9c1896b909aad05" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 35285 } } peers { permanent_uuid: "229332a1c0b84d9bb593579ac73af6a8" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36867 } }
I20260812 02:38:13.714918 21220 raft_consensus.cc:374] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:38:13.715003 21219 ts_tablet_manager.cc:1428] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260812 02:38:13.715003 21220 raft_consensus.cc:729] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75147e8e2c864cbea9c1896b909aad05, State: Initialized, Role: FOLLOWER
I20260812 02:38:13.715392 20996 heartbeater.cc:503] Master 127.3.127.190:33109 was elected leader, sending a full tablet report...
I20260812 02:38:13.715443 21220 consensus_queue.cc:260] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e83fd01a074b0e94a3af94a3c0708b" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37853 } } peers { permanent_uuid: "75147e8e2c864cbea9c1896b909aad05" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 35285 } } peers { permanent_uuid: "229332a1c0b84d9bb593579ac73af6a8" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36867 } }
I20260812 02:38:13.715185 21223 consensus_queue.cc:260] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e83fd01a074b0e94a3af94a3c0708b" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37853 } } peers { permanent_uuid: "75147e8e2c864cbea9c1896b909aad05" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 35285 } } peers { permanent_uuid: "229332a1c0b84d9bb593579ac73af6a8" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36867 } }
I20260812 02:38:13.716140 20781 heartbeater.cc:503] Master 127.3.127.190:33109 was elected leader, sending a full tablet report...
I20260812 02:38:13.716528 21220 ts_tablet_manager.cc:1428] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260812 02:38:13.717012 21223 ts_tablet_manager.cc:1428] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260812 02:38:13.717326 21193 heartbeater.cc:503] Master 127.3.127.190:33109 was elected leader, sending a full tablet report...
I20260812 02:38:13.874300 21230 raft_consensus.cc:482] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:38:13.874531 21230 raft_consensus.cc:504] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e83fd01a074b0e94a3af94a3c0708b" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37853 } } peers { permanent_uuid: "75147e8e2c864cbea9c1896b909aad05" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 35285 } } peers { permanent_uuid: "229332a1c0b84d9bb593579ac73af6a8" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36867 } }
I20260812 02:38:13.875797 21230 leader_election.cc:288] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69e83fd01a074b0e94a3af94a3c0708b (127.3.127.131:37853), 229332a1c0b84d9bb593579ac73af6a8 (127.3.127.130:36867)
I20260812 02:38:13.880756 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:13.876916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:46725 (local address 127.3.127.131:37853)
0812 02:38:13.877039 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:13.877042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:13.877056 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:38:13.877115 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:13.877118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:13.877250 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:13.877326 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:38:13.877750 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.878554 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.879747 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.880195 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.880317 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:13.880387 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 02:38:13.880538 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 02:38:13.880613 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 02:38:13.880786 21234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:13.876086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:37853 (local address 127.3.127.129:46725)
0812 02:38:13.876439 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:13.876455 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:13.876573 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:13.877468 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:13.877471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:13.877493 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:38:13.877618 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:13.877623 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.878718 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.878721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:13.879602 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:13.879614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.880318 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.880321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:13.880413 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:13.880440 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 02:38:13.880525 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 02:38:13.880604 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":26,"thread_start_us":152,"threads_started":1}
I20260812 02:38:13.881433 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:13.876751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:57313 (local address 127.3.127.130:36867)
0812 02:38:13.876880 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:13.876885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:13.877107 (+ 222us) server_negotiation.cc:407] Connection header received
0812 02:38:13.877300 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:13.877303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:13.877356 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:13.877401 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:38:13.877828 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.878553 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.880146 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.880687 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.880804 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:13.880854 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 02:38:13.881222 (+ 368us) server_negotiation.cc:299] Negotiation successful
0812 02:38:13.881309 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 02:38:13.881812 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:13.876566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:36867 (local address 127.3.127.129:57313)
0812 02:38:13.876993 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:13.877005 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:38:13.877139 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:38:13.877557 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:38:13.877560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:38:13.877579 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:38:13.877717 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:13.877722 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.878682 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.878686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:13.880003 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:38:13.880016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.880921 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.880925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:38:13.881033 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:13.881053 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 02:38:13.881589 (+ 536us) client_negotiation.cc:240] Negotiation successful
0812 02:38:13.881674 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":1,"thread_start_us":114,"threads_started":1}
I20260812 02:38:13.881937 21126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233c5ed4b52649a58619e5ab8c78b1d6" candidate_uuid: "75147e8e2c864cbea9c1896b909aad05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69e83fd01a074b0e94a3af94a3c0708b" is_pre_election: true
I20260812 02:38:13.882308 21126 raft_consensus.cc:2454] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75147e8e2c864cbea9c1896b909aad05 in term 0.
I20260812 02:38:13.882412 20917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233c5ed4b52649a58619e5ab8c78b1d6" candidate_uuid: "75147e8e2c864cbea9c1896b909aad05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "229332a1c0b84d9bb593579ac73af6a8" is_pre_election: true
I20260812 02:38:13.882746 20917 raft_consensus.cc:2454] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75147e8e2c864cbea9c1896b909aad05 in term 0.
I20260812 02:38:13.882828 20618 leader_election.cc:302] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69e83fd01a074b0e94a3af94a3c0708b, 75147e8e2c864cbea9c1896b909aad05; no voters:
I20260812 02:38:13.883159 21230 raft_consensus.cc:2781] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:38:13.883266 21230 raft_consensus.cc:482] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:38:13.883322 21230 raft_consensus.cc:3037] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:38:13.884473 21230 raft_consensus.cc:504] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e83fd01a074b0e94a3af94a3c0708b" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37853 } } peers { permanent_uuid: "75147e8e2c864cbea9c1896b909aad05" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 35285 } } peers { permanent_uuid: "229332a1c0b84d9bb593579ac73af6a8" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36867 } }
I20260812 02:38:13.884837 21230 leader_election.cc:288] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05 [CANDIDATE]: Term 1 election: Requested vote from peers 69e83fd01a074b0e94a3af94a3c0708b (127.3.127.131:37853), 229332a1c0b84d9bb593579ac73af6a8 (127.3.127.130:36867)
I20260812 02:38:13.885284 21126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233c5ed4b52649a58619e5ab8c78b1d6" candidate_uuid: "75147e8e2c864cbea9c1896b909aad05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69e83fd01a074b0e94a3af94a3c0708b"
I20260812 02:38:13.885349 20917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233c5ed4b52649a58619e5ab8c78b1d6" candidate_uuid: "75147e8e2c864cbea9c1896b909aad05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "229332a1c0b84d9bb593579ac73af6a8"
I20260812 02:38:13.885419 21126 raft_consensus.cc:3037] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:38:13.885488 20917 raft_consensus.cc:3037] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8 [term 0 FOLLOWER]: Advancing to term 1
W20260812 02:38:13.886049 21194 tablet.cc:2369] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:38:13.886684 20917 raft_consensus.cc:2454] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75147e8e2c864cbea9c1896b909aad05 in term 1.
I20260812 02:38:13.886783 21126 raft_consensus.cc:2454] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75147e8e2c864cbea9c1896b909aad05 in term 1.
I20260812 02:38:13.887004 20618 leader_election.cc:302] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 229332a1c0b84d9bb593579ac73af6a8, 75147e8e2c864cbea9c1896b909aad05; no voters:
I20260812 02:38:13.887223 21230 raft_consensus.cc:2781] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:38:13.887439 21230 raft_consensus.cc:686] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05 [term 1 LEADER]: Becoming Leader. State: Replica: 75147e8e2c864cbea9c1896b909aad05, State: Running, Role: LEADER
I20260812 02:38:13.887779 21230 consensus_queue.cc:237] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e83fd01a074b0e94a3af94a3c0708b" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37853 } } peers { permanent_uuid: "75147e8e2c864cbea9c1896b909aad05" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 35285 } } peers { permanent_uuid: "229332a1c0b84d9bb593579ac73af6a8" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36867 } }
I20260812 02:38:13.891237 20496 catalog_manager.cc:5495] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75147e8e2c864cbea9c1896b909aad05 (127.3.127.129). New cstate: current_term: 1 leader_uuid: "75147e8e2c864cbea9c1896b909aad05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e83fd01a074b0e94a3af94a3c0708b" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75147e8e2c864cbea9c1896b909aad05" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 35285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "229332a1c0b84d9bb593579ac73af6a8" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36867 } health_report { overall_health: UNKNOWN } } }
W20260812 02:38:13.907114 20999 tablet.cc:2369] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:38:13.925529 20782 tablet.cc:2369] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:38:13.935739 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:13.932083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49988 (local address 127.3.127.129:35285)
0812 02:38:13.932225 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 02:38:13.932229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:38:13.932243 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:38:13.932371 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:38:13.932373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:38:13.932433 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:38:13.932524 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:38:13.932802 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.933336 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.934278 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:38:13.934505 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:38:13.934588 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:38:13.934918 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:38:13.935371 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:38:13.935461 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 02:38:13.935504 (+ 43us) server_negotiation.cc:299] Negotiation successful
0812 02:38:13.935560 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 02:38:13.942226 20917 raft_consensus.cc:1261] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8 [term 1 FOLLOWER]: Refusing update from remote peer 75147e8e2c864cbea9c1896b909aad05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:38:13.942246 21126 raft_consensus.cc:1261] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b [term 1 FOLLOWER]: Refusing update from remote peer 75147e8e2c864cbea9c1896b909aad05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:38:13.943049 21238 consensus_queue.cc:1035] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69e83fd01a074b0e94a3af94a3c0708b" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:38:13.943281 21230 consensus_queue.cc:1035] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "229332a1c0b84d9bb593579ac73af6a8" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:38:13.961838 20495 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:38:13.962018 20495 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:38:13.964570 20495 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3cdc833159f41f485d16670c93bbce1: Sending DeleteTablet for 3 replicas of tablet 233c5ed4b52649a58619e5ab8c78b1d6
I20260812 02:38:13.965466 20691 tablet_service.cc:1514] Processing DeleteTablet for tablet 233c5ed4b52649a58619e5ab8c78b1d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:38:13 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:49978
I20260812 02:38:13.965816 21100 tablet_service.cc:1514] Processing DeleteTablet for tablet 233c5ed4b52649a58619e5ab8c78b1d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:38:13 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:52056
I20260812 02:38:13.966074 20890 tablet_service.cc:1514] Processing DeleteTablet for tablet 233c5ed4b52649a58619e5ab8c78b1d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:38:13 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:57624
I20260812 02:38:13.966185 21258 tablet_replica.cc:330] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05: stopping tablet replica
I20260812 02:38:13.966616 21259 tablet_replica.cc:330] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b: stopping tablet replica
I20260812 02:38:13.966637 21258 raft_consensus.cc:2229] T 233c5ed4b52649a58619e5ab8c78b1d6 P 75147e8e2c864cbea9c1896b909aad05 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:38:13.966715 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 20593
I20260812 02:38:13.966935 21259 raft_consensus.cc:2229] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:38:13.967447 21260 tablet_replica.cc:330] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8: stopping tablet replica
I20260812 02:38:13.967758 21260 raft_consensus.cc:2229] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:38:13.968052 21260 raft_consensus.cc:2258] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:38:13.968274 21259 raft_consensus.cc:2258] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:38:13.971406 21260 ts_tablet_manager.cc:1905] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:38:13.973824 21260 ts_tablet_manager.cc:1918] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:38:13.973932 21260 log.cc:1196] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/233c5ed4b52649a58619e5ab8c78b1d6
I20260812 02:38:13.974233 21260 ts_tablet_manager.cc:1939] T 233c5ed4b52649a58619e5ab8c78b1d6 P 229332a1c0b84d9bb593579ac73af6a8: Deleting consensus metadata
I20260812 02:38:13.974818 21259 ts_tablet_manager.cc:1905] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:38:13.975517 20478 catalog_manager.cc:4841] TS 229332a1c0b84d9bb593579ac73af6a8 (127.3.127.130:36867): tablet 233c5ed4b52649a58619e5ab8c78b1d6 (table test-table [id=2a3548122bd84bb4987a01931e6d62e2]) successfully deleted
I20260812 02:38:13.977799 21259 ts_tablet_manager.cc:1918] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:38:13.977929 21259 log.cc:1196] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/233c5ed4b52649a58619e5ab8c78b1d6
I20260812 02:38:13.978255 21259 ts_tablet_manager.cc:1939] T 233c5ed4b52649a58619e5ab8c78b1d6 P 69e83fd01a074b0e94a3af94a3c0708b: Deleting consensus metadata
I20260812 02:38:13.979935 20478 catalog_manager.cc:4841] TS 69e83fd01a074b0e94a3af94a3c0708b (127.3.127.131:37853): tablet 233c5ed4b52649a58619e5ab8c78b1d6 (table test-table [id=2a3548122bd84bb4987a01931e6d62e2]) successfully deleted
W20260812 02:38:13.981900 20475 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.127.129:35285 (error 108)
I20260812 02:38:13.982000 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 20789
I20260812 02:38:13.991914 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:38:13.991578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:35285 (local address 127.0.0.1:50002)
0812 02:38:13.991736 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 02:38:13.991820 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.129:35285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260812 02:38:13.992138 20475 catalog_manager.cc:4584] TS 75147e8e2c864cbea9c1896b909aad05 (127.3.127.129:35285): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 233c5ed4b52649a58619e5ab8c78b1d6: Network error: Client connection negotiation failed: client connection to 127.3.127.129:35285: connect: Connection refused (error 111)
I20260812 02:38:13.997450 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 21003
I20260812 02:38:14.013571 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 20444
2026-08-12T02:38:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2663 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:38:14 dist-test-slave-rmhg krb5kdc[21266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:38:14 dist-test-slave-rmhg krb5kdc[21266](info): set up 2 sockets
Aug 12 02:38:14 dist-test-slave-rmhg krb5kdc[21266](info): commencing operation
krb5kdc: starting...
W20260812 02:38:15.198650 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:38:15 dist-test-slave-rmhg krb5kdc[21266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:38:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:38:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.127.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.127.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.127.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/rangeradmin_127.3.127.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.127.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/rangeradmin_127.3.127.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.127.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.127.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.127.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/rangerlookup_127.3.127.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.127.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/rangerlookup_127.3.127.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.127.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.127.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.127.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/HTTP_127.3.127.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.127.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/HTTP_127.3.127.148@KRBTEST.COM.keytab.
I20260812 02:38:15.384491 3582 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task9rbVbc/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/postgres -l logfile start
2026-08-12 02:38:18.030 UTC [21318] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 02:38:18.030 UTC [21318] LOG: listening on IPv4 address "127.3.127.148", port 59387
2026-08-12 02:38:18.033 UTC [21318] LOG: listening on Unix socket "/tmp/.s.PGSQL.59387"
2026-08-12 02:38:18.055 UTC [21326] LOG: database system was shut down at 2026-08-12 02:38:16 UTC
2026-08-12 02:38:18.059 UTC [21318] LOG: database system is ready to accept connections
I20260812 02:38:19.136879 3582 mini_postgres.cc:96] Postgres bound to 59387
2026-08-12 02:38:19.146 UTC [21337] FATAL: database "slave" does not exist
127.3.127.148:59387 - accepting connections
I20260812 02:38:19.147006 3582 mini_ranger.cc:161] Starting Ranger...
I20260812 02:38:19.165098 3582 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 02:38:19.668480 3582 mini_ranger.cc:87] Created ranger Postgres database
I20260812 02:38:19.668565 3582 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ranger-admin
2026-08-12 02:38:20,006 [I] DB FLAVOR :POSTGRES
2026-08-12 02:38:20,006 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 02:38:20,006 [I] Checking connection..
2026-08-12 02:38:20,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 02:38:20,280 [I] Checking connection passed.
2026-08-12 02:38:20,280 [I] --------- Verifying version history table ---------
2026-08-12 02:38:20,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 02:38:20,569 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 02:38:20,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 02:38:20,844 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 02:38:20,844 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 02:38:20,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 02:38:21.119 UTC [21429] WARNING: there is no transaction in progress
2026-08-12 02:38:21,129 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 02:38:21,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 02:38:21,413 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 02:38:21,413 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 02:38:21,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 02:38:21,690 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 02:38:21,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rmhg.c.gcp-upstream.internal','N') ;"
2026-08-12 02:38:22,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 02:38:22,484 [I] Table x_portal_user does not exist in database ranger
2026-08-12 02:38:22,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 02:38:22,756 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 02:38:22,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 02:38:23,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 02:38:23,290 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 02:38:23,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 02:38:24.332 UTC [21595] WARNING: there is no transaction in progress
2026-08-12 02:38:24.350 UTC [21595] WARNING: there is no transaction in progress
2026-08-12 02:38:24.367 UTC [21595] WARNING: there is no transaction in progress
2026-08-12 02:38:24.392 UTC [21595] WARNING: there is no transaction in progress
2026-08-12 02:38:24.410 UTC [21595] WARNING: there is no transaction in progress
2026-08-12 02:38:24.428 UTC [21595] WARNING: there is no transaction in progress
2026-08-12 02:38:24.487 UTC [21595] WARNING: there is no transaction in progress
2026-08-12 02:38:24.494 UTC [21595] WARNING: there is no transaction in progress
2026-08-12 02:38:24.509 UTC [21595] WARNING: there is no transaction in progress
2026-08-12 02:38:24.520 UTC [21595] WARNING: there is no transaction in progress
2026-08-12 02:38:24.530 UTC [21595] WARNING: there is no transaction in progress
2026-08-12 02:38:24.543 UTC [21595] WARNING: there is no transaction in progress
2026-08-12 02:38:24.553 UTC [21595] WARNING: there is no transaction in progress
2026-08-12 02:38:24.560 UTC [21595] WARNING: there is no transaction in progress
2026-08-12 02:38:24.567 UTC [21595] WARNING: there is no transaction in progress
2026-08-12 02:38:24.575 UTC [21595] WARNING: there is no transaction in progress
2026-08-12 02:38:25,264 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 02:38:25,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 02:38:25,516 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 02:38:25,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 02:38:25,795 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 02:38:25,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 02:38:26,050 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 02:38:26,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 02:38:26,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 02:38:26,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rmhg.c.gcp-upstream.internal';"
2026-08-12 02:38:26,850 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 02:38:26,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 02:38:27,113 [I] DB_PATCHES have already been applied
I20260812 02:38:27.119416 3582 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ranger-admin:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task9rbVbc/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 02:38:27.119499 3582 mini_ranger.cc:193] Using host: 127.3.127.148
I20260812 02:38:27.124603 3582 mini_ranger.cc:227] Ranger admin URL: http://127.3.127.148:38877
Aug 12, 2026 2:38:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:38:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:38:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 2:38:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 2:38:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 2:38:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 2:38:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 02:38:28.400913 3582 mini_ranger.cc:160] Time spent starting Ranger: real 9.254s user 0.008s sys 0.005s
Aug 12, 2026 2:38:51 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 2:38:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 12, 2026 2:38:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 2:38:51 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 2:38:52 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 02:38:53.154923 3582 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.127.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.127.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.127.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/rangerkms_127.3.127.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.127.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/rangerkms_127.3.127.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.127.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.127.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.127.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/HTTP_127.3.127.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.127.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/HTTP_127.3.127.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 02:38:53 dist-test-slave-rmhg krb5kdc[21266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502333, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 02:38:53.246246 3582 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 02:38:53.258029 3582 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 02:38:54.173964 3582 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 02:38:54.174067 3582 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ranger-kms
I20260812 02:38:54.174175 3582 mini_ranger_kms.cc:227] Using postgres at 127.3.127.148:59387
2026-08-12 02:38:54,630 [I] DB FLAVOR :POSTGRES
2026-08-12 02:38:54,630 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 02:38:54,630 [I] Checking connection
2026-08-12 02:38:54,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 02:38:54,920 [I] connection success
2026-08-12 02:38:54,920 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 02:38:54,920 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 02:38:54,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 02:38:55,184 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 02:38:55,185 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 02:38:55,185 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 02:38:55,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:59387/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 02:38:55,466 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 02:38:56.018968 3582 mini_ranger_kms.cc:310] Created kms service
I20260812 02:38:56.229753 3582 mini_ranger_kms.cc:326] Created kudu user
I20260812 02:38:56.276360 3582 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 02:38:56 dist-test-slave-rmhg krb5kdc[21266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502333, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
I20260812 02:38:56.770450 3582 mini_ranger_kms.cc:384] Added ranger policy
I20260812 02:38:56.770622 3582 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ranger-kms:/tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 02:38:56.770648 3582 mini_ranger_kms.cc:240] Using host: 127.3.127.148
I20260812 02:38:56.777020 3582 mini_ranger_kms.cc:276] Ranger KMS PID: 21875
I20260812 02:38:56.777316 3582 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.127.148:44503
Aug 12, 2026 2:38:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:38:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:38:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 2:38:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 2:38:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 2:38:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 02:38:59 dist-test-slave-rmhg krb5kdc[21266](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786502339, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.127.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:38:59.363416 3582 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.117s user 0.021s sys 0.415s
I20260812 02:38:59.363574 3582 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 02:38:59.363713 3582 mini_ranger_kms.cc:401] 127.3.127.148:44503/kms/v1/keys
Aug 12 02:38:59 dist-test-slave-rmhg krb5kdc[21266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502339, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.127.148@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
Aug 12 02:39:00 dist-test-slave-rmhg krb5kdc[21266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502339, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.127.148@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
Aug 12 02:39:02 dist-test-slave-rmhg krb5kdc[21266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:39:02.570398 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:37145
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:40821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:37145
--ranger_config_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.127.148:44503/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:39:02.742218 21947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:39:02.742532 21947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:39:02.742575 21947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:39:02.747278 21947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:39:02.747363 21947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:39:02.747381 21947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:39:02.747397 21947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:39:02.747411 21947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:39:02.753545 21947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:40821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.127.148:44503/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:37145
--ranger_config_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:37145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:39:02.754753 21947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:39:02.755856 21947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:39:02.767380 21959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:39:02.767504 21960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:39:02.767601 21947 server_base.cc:1034] running on GCE node
W20260812 02:39:02.767375 21963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:39:02.768357 21947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:39:02.769753 21947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:39:02.770957 21947 hybrid_clock.cc:648] HybridClock initialized: now 1786502342770935 us; error 39 us; skew 500 ppm
Aug 12 02:39:02 dist-test-slave-rmhg krb5kdc[21266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502342, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:39:02.774260 21947 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:39:02.775388 21947 webserver.cc:460] Webserver started at http://127.3.127.190:39043/ using document root <none> and password file <none>
I20260812 02:39:02.775976 21947 fs_manager.cc:359] Metadata directory not provided
I20260812 02:39:02.776047 21947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:39:02.776264 21947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:39:02 dist-test-slave-rmhg krb5kdc[21266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502342, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
I20260812 02:39:02.831295 21947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "84c89d48a0184900bdb27fd2a4611360"
format_stamp: "Formatted at 2026-08-12 02:39:02 on dist-test-slave-rmhg"
server_key: "b7ea4b4e0ec1f410d7f5fcf73236b1d9"
server_key_iv: "ac0f50990a07df85ae305ccc1cfd59f6"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:02.832172 21947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "84c89d48a0184900bdb27fd2a4611360"
format_stamp: "Formatted at 2026-08-12 02:39:02 on dist-test-slave-rmhg"
server_key: "b7ea4b4e0ec1f410d7f5fcf73236b1d9"
server_key_iv: "ac0f50990a07df85ae305ccc1cfd59f6"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:02.837988 21947 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 02:39:02.854373 21977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:02.855901 21947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:39:02.856061 21947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "84c89d48a0184900bdb27fd2a4611360"
format_stamp: "Formatted at 2026-08-12 02:39:02 on dist-test-slave-rmhg"
server_key: "b7ea4b4e0ec1f410d7f5fcf73236b1d9"
server_key_iv: "ac0f50990a07df85ae305ccc1cfd59f6"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:02.856202 21947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:39:02.876569 21947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:39:02.879912 21947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:39:02.880414 21947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:39:02.891708 21947 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:37145
I20260812 02:39:02.891780 22065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:37145 every 8 connection(s)
I20260812 02:39:02.892823 21947 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:39:02.896755 22067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:39:02.902107 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 21947
I20260812 02:39:02.902245 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:39:02.902560 3582 external_mini_cluster.cc:1428] Setting key 9dc0616424ebde3afddfd6dd181c9bf3
I20260812 02:39:02.905180 22067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360: Bootstrap starting.
I20260812 02:39:02.908293 22067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360: Neither blocks nor log segments found. Creating new log.
I20260812 02:39:02.909291 22067 log.cc:824] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360: Log is configured to *not* fsync() on all Append() calls
I20260812 02:39:02.911646 22067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360: No bootstrap required, opened a new log
Aug 12 02:39:02 dist-test-slave-rmhg krb5kdc[21266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:39:02.916273 22067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c89d48a0184900bdb27fd2a4611360" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37145 } }
I20260812 02:39:02.916675 22067 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:39:02.916739 22067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c89d48a0184900bdb27fd2a4611360, State: Initialized, Role: FOLLOWER
I20260812 02:39:02.917308 22067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c89d48a0184900bdb27fd2a4611360" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37145 } }
I20260812 02:39:02.917471 22067 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:39:02.917558 22067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:39:02.917659 22067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:39:02.918416 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:02.904267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37262 (local address 127.3.127.190:37145)
0812 02:39:02.904781 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:02.904797 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:02.904838 (+ 41us) server_negotiation.cc:407] Connection header received
0812 02:39:02.905783 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:02.905807 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:02.906196 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:02.906537 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:39:02.907390 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:02.908567 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:02.909519 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:02.910234 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:02.910390 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:02.913485 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:39:02.913498 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:39:02.913523 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:39:02.913555 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:39:02.915194 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:02.915634 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:02.915639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:02.915645 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:02.915772 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:02.916044 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:02.916047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:02.916048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:02.916466 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:39:02.916685 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:02.917319 (+ 634us) server_negotiation.cc:299] Negotiation successful
0812 02:39:02.917610 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":189,"threads_started":1}
I20260812 02:39:02.918887 22067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c89d48a0184900bdb27fd2a4611360" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37145 } }
I20260812 02:39:02.919278 22067 leader_election.cc:302] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84c89d48a0184900bdb27fd2a4611360; no voters:
I20260812 02:39:02.919623 22067 leader_election.cc:288] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:39:02.919806 22078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:39:02.920090 22078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360 [term 1 LEADER]: Becoming Leader. State: Replica: 84c89d48a0184900bdb27fd2a4611360, State: Running, Role: LEADER
I20260812 02:39:02.920423 22078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c89d48a0184900bdb27fd2a4611360" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37145 } }
I20260812 02:39:02.920940 22067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:39:02.921854 22078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84c89d48a0184900bdb27fd2a4611360. Latest consensus state: current_term: 1 leader_uuid: "84c89d48a0184900bdb27fd2a4611360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c89d48a0184900bdb27fd2a4611360" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37145 } } }
I20260812 02:39:02.921854 22083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84c89d48a0184900bdb27fd2a4611360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c89d48a0184900bdb27fd2a4611360" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37145 } } }
I20260812 02:39:02.922096 22083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360 [sys.catalog]: This master's current role is: LEADER
I20260812 02:39:02.922096 22078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360 [sys.catalog]: This master's current role is: LEADER
I20260812 02:39:02.922540 22094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:39:02.926095 22094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:39:02.926661 22067 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 02:39:02.939870 22094 catalog_manager.cc:1347] Generated new cluster ID: d383e40dba1e44788b1ebdc266392169
I20260812 02:39:02.939982 22094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:39:02.966086 22094 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:39:02.967708 22094 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:39:02.999563 22094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360: Generated new TSK 0
I20260812 02:39:03.000890 22094 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
I20260812 02:39:03.877322 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.127.148:44503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37145
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:40821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:39:04.074754 22141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:39:04.075248 22141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:39:04.075344 22141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:39:04.081995 22141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:39:04.082098 22141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:39:04.082190 22141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:39:04.090842 22141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:40821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.127.148:44503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:39:04.092839 22141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:39:04.094949 22141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:39:04.108808 22153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:39:04.108983 22141 server_base.cc:1034] running on GCE node
W20260812 02:39:04.109437 22158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:39:04.109510 22152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:39:04.110296 22141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:39:04.111267 22141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:39:04.112500 22141 hybrid_clock.cc:648] HybridClock initialized: now 1786502344112470 us; error 41 us; skew 500 ppm
Aug 12 02:39:04 dist-test-slave-rmhg krb5kdc[21266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502344, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:39:04.116592 22141 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:39:04.117913 22141 webserver.cc:460] Webserver started at http://127.3.127.129:39927/ using document root <none> and password file <none>
I20260812 02:39:04.118705 22141 fs_manager.cc:359] Metadata directory not provided
I20260812 02:39:04.118801 22141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:39:04.119052 22141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:39:04 dist-test-slave-rmhg krb5kdc[21266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502344, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
I20260812 02:39:04.138214 22141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "c8e0a0f12c6248e5af5f3ca69ca4271d"
format_stamp: "Formatted at 2026-08-12 02:39:04 on dist-test-slave-rmhg"
server_key: "cdad719b3d6ed220113d798da5b84b94"
server_key_iv: "fff66691513885b4479992eca71a6261"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:04.139107 22141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "c8e0a0f12c6248e5af5f3ca69ca4271d"
format_stamp: "Formatted at 2026-08-12 02:39:04 on dist-test-slave-rmhg"
server_key: "cdad719b3d6ed220113d798da5b84b94"
server_key_iv: "fff66691513885b4479992eca71a6261"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:04.145330 22141 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 02:39:04.177676 22172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:04.179436 22141 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.003s sys 0.000s
I20260812 02:39:04.179589 22141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "c8e0a0f12c6248e5af5f3ca69ca4271d"
format_stamp: "Formatted at 2026-08-12 02:39:04 on dist-test-slave-rmhg"
server_key: "cdad719b3d6ed220113d798da5b84b94"
server_key_iv: "fff66691513885b4479992eca71a6261"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:04.179749 22141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:39:04.201381 22141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:39:04.205372 22141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:39:04.205660 22141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:39:04.206462 22141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:39:04.207682 22141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:39:04.207806 22141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 02:39:04.207948 22141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:39:04.208045 22141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:04.226240 22141 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:40299
I20260812 02:39:04.226461 22343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:40299 every 8 connection(s)
I20260812 02:39:04.228029 22141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:39:04.233475 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 22141
I20260812 02:39:04.233635 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:39:04.233997 3582 external_mini_cluster.cc:1428] Setting key e7875bb11744f80a3b1753a78f9261be
Aug 12 02:39:04 dist-test-slave-rmhg krb5kdc[21266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502344, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:39:04.247349 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:04.230553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:42699 (local address 127.3.127.190:37145)
0812 02:39:04.230857 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:04.230862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:04.231944 (+ 1082us) server_negotiation.cc:407] Connection header received
0812 02:39:04.233101 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:04.233105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:04.233199 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:04.233290 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:39:04.235628 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:04.236387 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:04.238200 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:04.238585 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:04.238684 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:04.242291 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:39:04.242296 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:39:04.242299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:39:04.242334 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:39:04.244352 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:04.244972 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:04.244977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:04.244978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:04.245095 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:04.245676 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:04.245680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:04.245682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:04.245973 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:39:04.246092 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:04.246958 (+ 866us) server_negotiation.cc:299] Negotiation successful
0812 02:39:04.247140 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":106,"threads_started":1}
I20260812 02:39:04.249619 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:04.231096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:37145 (local address 127.3.127.129:42699)
0812 02:39:04.231728 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:04.231790 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 02:39:04.232827 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:39:04.233680 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:39:04.233691 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:39:04.234133 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:39:04.235424 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:04.235449 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:04.236587 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:04.236590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:04.238037 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:04.238051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:04.238958 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:04.238963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:04.239213 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:04.240023 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:39:04.240049 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:39:04.242030 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:39:04.244497 (+ 2467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:39:04.244505 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:39:04.244535 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:39:04.244839 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:39:04.245262 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:39:04.245265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:39:04.245267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:39:04.245514 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:39:04.246196 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:39:04.246204 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:39:04.246611 (+ 407us) client_negotiation.cc:769] Sending connection context
0812 02:39:04.247906 (+ 1295us) client_negotiation.cc:240] Negotiation successful
0812 02:39:04.248343 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":221,"threads_started":1}
WARNING: no policy specified for kudu/127.3.127.130@KRBTEST.COM; defaulting to no policy
I20260812 02:39:04.251497 22345 heartbeater.cc:347] Connected to a master server at 127.3.127.190:37145
I20260812 02:39:04.251784 22345 heartbeater.cc:464] Registering TS with master...
I20260812 02:39:04.252400 22345 heartbeater.cc:511] Master 127.3.127.190:37145 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.130@KRBTEST.COM" created.
I20260812 02:39:04.254406 22006 ts_manager.cc:194] Registered new tserver with Master: c8e0a0f12c6248e5af5f3ca69ca4271d (127.3.127.129:40299)
I20260812 02:39:04.255726 22006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:42699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
I20260812 02:39:04.277269 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.127.148:44503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37145
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:40821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:39:04 dist-test-slave-rmhg krb5kdc[21266](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786502344, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 02:39:04.439059 22359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:39:04.439494 22359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:39:04.439571 22359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:39:04.445839 22359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:39:04.445935 22359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:39:04.446038 22359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:39:04.454523 22359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:40821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.127.148:44503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:39:04.456279 22359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:39:04.457949 22359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:39:04.472841 22370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:39:04.472841 22366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:39:04.472841 22367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:39:04.473335 22359 server_base.cc:1034] running on GCE node
I20260812 02:39:04.473855 22359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:39:04.474545 22359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:39:04.475728 22359 hybrid_clock.cc:648] HybridClock initialized: now 1786502344475712 us; error 32 us; skew 500 ppm
Aug 12 02:39:04 dist-test-slave-rmhg krb5kdc[21266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502344, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:39:04.478948 22359 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:39:04.480150 22359 webserver.cc:460] Webserver started at http://127.3.127.130:41189/ using document root <none> and password file <none>
I20260812 02:39:04.480873 22359 fs_manager.cc:359] Metadata directory not provided
I20260812 02:39:04.481042 22359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:39:04.481408 22359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:39:04 dist-test-slave-rmhg krb5kdc[21266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502344, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
I20260812 02:39:04.498437 22359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "8753a9bda42148a8863d664dfee6281c"
format_stamp: "Formatted at 2026-08-12 02:39:04 on dist-test-slave-rmhg"
server_key: "94cbdbf185bf7b8cb38b0b0e04419ec5"
server_key_iv: "87100b93aa66c0c08dd1f93ad7e99693"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:04.499413 22359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "8753a9bda42148a8863d664dfee6281c"
format_stamp: "Formatted at 2026-08-12 02:39:04 on dist-test-slave-rmhg"
server_key: "94cbdbf185bf7b8cb38b0b0e04419ec5"
server_key_iv: "87100b93aa66c0c08dd1f93ad7e99693"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:04.511030 22359 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.004s sys 0.000s
I20260812 02:39:04.539003 22380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:04.541654 22359 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 02:39:04.541934 22359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "8753a9bda42148a8863d664dfee6281c"
format_stamp: "Formatted at 2026-08-12 02:39:04 on dist-test-slave-rmhg"
server_key: "94cbdbf185bf7b8cb38b0b0e04419ec5"
server_key_iv: "87100b93aa66c0c08dd1f93ad7e99693"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:04.542228 22359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:39:04.581382 22359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:39:04.585707 22359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:39:04.586016 22359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:39:04.586961 22359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:39:04.588284 22359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:39:04.588368 22359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:04.588440 22359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:39:04.588475 22359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:04.605185 22359 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:43983
I20260812 02:39:04.607025 22562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:43983 every 8 connection(s)
I20260812 02:39:04.607610 22359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:39:04.609484 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 22359
I20260812 02:39:04.609640 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:39:04.609977 3582 external_mini_cluster.cc:1428] Setting key bee1f1dbaf9551a699a121242e6bb4ef
Aug 12 02:39:04 dist-test-slave-rmhg krb5kdc[21266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502344, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:39:04.627260 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:04.611940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:54437 (local address 127.3.127.190:37145)
0812 02:39:04.612075 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:04.612079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:04.613683 (+ 1604us) server_negotiation.cc:407] Connection header received
0812 02:39:04.615089 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:04.615093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:04.615149 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:04.615236 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:39:04.616973 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:04.617690 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:04.619445 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:04.619694 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:04.619781 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:04.623297 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:39:04.623301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:39:04.623304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:39:04.623336 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:39:04.624920 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:04.625567 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:04.625571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:04.625573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:04.625657 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:04.626062 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:04.626065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:04.626066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:04.626273 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:39:04.626360 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:04.626933 (+ 573us) server_negotiation.cc:299] Negotiation successful
0812 02:39:04.627080 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 02:39:04.628289 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:04.610805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:37145 (local address 127.3.127.130:54437)
0812 02:39:04.613514 (+ 2709us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:04.613579 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 02:39:04.614853 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:39:04.615518 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:39:04.615529 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:39:04.615974 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:39:04.616707 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:04.616727 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:04.617839 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:04.617843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:04.619296 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:04.619308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:04.619863 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:04.619867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:04.620132 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:04.620892 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:39:04.620915 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:39:04.623050 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:39:04.625083 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:39:04.625089 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:39:04.625104 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:39:04.625451 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:39:04.625760 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:39:04.625763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:39:04.625765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:39:04.625937 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:39:04.626383 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:39:04.626391 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:39:04.626676 (+ 285us) client_negotiation.cc:769] Sending connection context
0812 02:39:04.626849 (+ 173us) client_negotiation.cc:240] Negotiation successful
0812 02:39:04.627235 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2443,"thread_start_us":157,"threads_started":1}
I20260812 02:39:04.629601 22564 heartbeater.cc:347] Connected to a master server at 127.3.127.190:37145
I20260812 02:39:04.629864 22564 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.127.131@KRBTEST.COM; defaulting to no policy
I20260812 02:39:04.630421 22564 heartbeater.cc:511] Master 127.3.127.190:37145 requested a full tablet report, sending...
I20260812 02:39:04.631564 22006 ts_manager.cc:194] Registered new tserver with Master: 8753a9bda42148a8863d664dfee6281c (127.3.127.130:43983)
I20260812 02:39:04.632390 22006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:54437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
I20260812 02:39:04.658346 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.127.148:44503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37145
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:40821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:39:04 dist-test-slave-rmhg krb5kdc[21266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502344, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
W20260812 02:39:04.863646 22576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:39:04.864070 22576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:39:04.864141 22576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:39:04.870486 22576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:39:04.870633 22576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:39:04.870747 22576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:39:04.879119 22576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:40821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.127.148:44503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:39:04.880950 22576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:39:04.882493 22576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:39:04.898351 22586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:39:04.898644 22576 server_base.cc:1034] running on GCE node
W20260812 02:39:04.898725 22584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:39:04.901369 22583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:39:04.902153 22576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:39:04.903110 22576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:39:04.906924 22576 hybrid_clock.cc:648] HybridClock initialized: now 1786502344906820 us; error 113 us; skew 500 ppm
Aug 12 02:39:04 dist-test-slave-rmhg krb5kdc[21266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502344, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:39:04.912492 22576 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:39:04.913853 22576 webserver.cc:460] Webserver started at http://127.3.127.131:46447/ using document root <none> and password file <none>
I20260812 02:39:04.914690 22576 fs_manager.cc:359] Metadata directory not provided
I20260812 02:39:04.914846 22576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:39:04.915205 22576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:39:04 dist-test-slave-rmhg krb5kdc[21266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502344, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
I20260812 02:39:04.954485 22576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "239139e741c84a7cbe4302f9e4b827d3"
format_stamp: "Formatted at 2026-08-12 02:39:04 on dist-test-slave-rmhg"
server_key: "6be3f761f57cf8d8f18af50975b014c4"
server_key_iv: "5e984f1e9461633ab4c4cb1269980709"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:04.957137 22576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "239139e741c84a7cbe4302f9e4b827d3"
format_stamp: "Formatted at 2026-08-12 02:39:04 on dist-test-slave-rmhg"
server_key: "6be3f761f57cf8d8f18af50975b014c4"
server_key_iv: "5e984f1e9461633ab4c4cb1269980709"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:04.962237 22576 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260812 02:39:04.983561 22604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:04.985324 22576 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:39:04.985567 22576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "239139e741c84a7cbe4302f9e4b827d3"
format_stamp: "Formatted at 2026-08-12 02:39:04 on dist-test-slave-rmhg"
server_key: "6be3f761f57cf8d8f18af50975b014c4"
server_key_iv: "5e984f1e9461633ab4c4cb1269980709"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:04.985807 22576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:39:05.006592 22576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:39:05.010573 22576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:39:05.010792 22576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:39:05.011590 22576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:39:05.012741 22576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:39:05.012811 22576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:05.012928 22576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:39:05.012964 22576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:05.030827 22576 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:36327
I20260812 02:39:05.030913 22794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:36327 every 8 connection(s)
I20260812 02:39:05.032246 22576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:39:05.034497 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 22576
I20260812 02:39:05.034633 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:39:05.034986 3582 external_mini_cluster.cc:1428] Setting key 41c9dd4bdf56d2f2dba0df235f9a3eee
Aug 12 02:39:05 dist-test-slave-rmhg krb5kdc[21266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502344, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:39:05.051460 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:05.036557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:60907 (local address 127.3.127.190:37145)
0812 02:39:05.036671 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:05.036676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:05.036691 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:39:05.037566 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:05.037569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:05.037627 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:05.037703 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:39:05.039650 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.040452 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.042361 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.042703 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.042818 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:05.046201 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:39:05.046216 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:39:05.046219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:39:05.046258 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:39:05.048294 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:05.049053 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:05.049057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:05.049059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:05.049191 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:05.049776 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:05.049780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:05.049782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:05.050040 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:39:05.050138 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:05.051014 (+ 876us) server_negotiation.cc:299] Negotiation successful
0812 02:39:05.051193 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 02:39:05.052393 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:05.035332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:37145 (local address 127.3.127.131:60907)
0812 02:39:05.036012 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:05.036073 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 02:39:05.037309 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:39:05.038074 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:39:05.038084 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:39:05.038559 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:39:05.039435 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:05.039462 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.040648 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.040651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:05.042190 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:05.042214 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.042885 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.042889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:05.043113 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:05.043834 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:39:05.043858 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:39:05.045918 (+ 2060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:39:05.048450 (+ 2532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:39:05.048458 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:39:05.048481 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:39:05.048924 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:39:05.049336 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:39:05.049339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:39:05.049341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:39:05.049631 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:39:05.050275 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:39:05.050285 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:39:05.050669 (+ 384us) client_negotiation.cc:769] Sending connection context
0812 02:39:05.050891 (+ 222us) client_negotiation.cc:240] Negotiation successful
0812 02:39:05.051274 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":234,"threads_started":1}
I20260812 02:39:05.053728 22795 heartbeater.cc:347] Connected to a master server at 127.3.127.190:37145
I20260812 02:39:05.054020 22795 heartbeater.cc:464] Registering TS with master...
I20260812 02:39:05.054677 22795 heartbeater.cc:511] Master 127.3.127.190:37145 requested a full tablet report, sending...
I20260812 02:39:05.055984 22006 ts_manager.cc:194] Registered new tserver with Master: 239139e741c84a7cbe4302f9e4b827d3 (127.3.127.131:36327)
I20260812 02:39:05.056856 22006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:60907
I20260812 02:39:05.062604 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:39:05.075863 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:05.065698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37276 (local address 127.3.127.190:37145)
0812 02:39:05.065845 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:05.065849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:05.066204 (+ 355us) server_negotiation.cc:407] Connection header received
0812 02:39:05.066370 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:05.066373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:05.066426 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:05.066513 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:39:05.067500 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.068345 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.069722 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.070056 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.070162 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:05.071954 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:39:05.071958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:39:05.071961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:39:05.071997 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:39:05.073830 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:05.074378 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:05.074382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:05.074384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:05.074515 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:05.074846 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:05.074849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:05.074851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:05.075090 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:39:05.075224 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:05.075550 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 02:39:05.075689 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 02:39:05.080080 22006 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:39:05.084784 22006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:39:05.104939 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:05.096369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:40299 (local address 127.0.0.1:60916)
0812 02:39:05.096897 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:05.096954 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 02:39:05.097091 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:39:05.098771 (+ 1680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:39:05.098778 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:39:05.098807 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:39:05.098989 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:05.098998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.101221 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.101225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:05.102699 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:05.102712 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.104264 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.104269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:05.104447 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:05.104520 (+ 73us) client_negotiation.cc:769] Sending connection context
0812 02:39:05.104637 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 02:39:05.104779 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":144,"threads_started":1}
I20260812 02:39:05.106015 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:05.097389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60916 (local address 127.3.127.129:40299)
0812 02:39:05.097681 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:05.097690 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:05.097715 (+ 25us) server_negotiation.cc:407] Connection header received
0812 02:39:05.097799 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:05.097804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:05.098039 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:05.098231 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:39:05.099259 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.101021 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.102867 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.103858 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.103982 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:05.104175 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:05.105682 (+ 1507us) server_negotiation.cc:299] Negotiation successful
0812 02:39:05.105812 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":90,"threads_started":1}
I20260812 02:39:05.108067 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:05.097851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:36327 (local address 127.0.0.1:57408)
0812 02:39:05.100726 (+ 2875us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:05.100741 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:39:05.100838 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:39:05.101649 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:39:05.101652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:39:05.101680 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:39:05.101788 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:05.101794 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.104106 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.104110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:05.105346 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:05.105358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.107552 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.107556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:05.107727 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:05.107757 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 02:39:05.107837 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 02:39:05.107919 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2809,"thread_start_us":187,"threads_started":1}
I20260812 02:39:05.108696 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:05.098384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57408 (local address 127.3.127.131:36327)
0812 02:39:05.100493 (+ 2109us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:05.100503 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:05.100935 (+ 432us) server_negotiation.cc:407] Connection header received
0812 02:39:05.101023 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:05.101029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:05.101289 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:05.101485 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:39:05.102782 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.103962 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.105573 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.107405 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.107534 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:05.107694 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:05.108389 (+ 695us) server_negotiation.cc:299] Negotiation successful
0812 02:39:05.108516 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2012,"thread_start_us":143,"threads_started":1}
I20260812 02:39:05.109750 22219 tablet_service.cc:1467] Processing CreateTablet for tablet d97477afaa5942dfb1213031cfa83b2e (DEFAULT_TABLE table=test-table [id=7a5794c981b04bcf81965a4735505c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:39:05.111140 22219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d97477afaa5942dfb1213031cfa83b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:39:05.112448 22678 tablet_service.cc:1467] Processing CreateTablet for tablet d97477afaa5942dfb1213031cfa83b2e (DEFAULT_TABLE table=test-table [id=7a5794c981b04bcf81965a4735505c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:39:05.113790 22678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d97477afaa5942dfb1213031cfa83b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:39:05.121327 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:05.099711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:43983 (local address 127.0.0.1:47250)
0812 02:39:05.104511 (+ 4800us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:05.104528 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:39:05.104637 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:39:05.106224 (+ 1587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:39:05.106227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:39:05.106250 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:39:05.106346 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:05.106353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.108443 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.108447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:05.109665 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:05.109677 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.120724 (+ 11047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.120730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:05.120869 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:05.120911 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 02:39:05.121018 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 02:39:05.121128 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4706,"thread_start_us":90,"threads_started":1}
I20260812 02:39:05.122071 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:05.099889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.3.127.130:43983)
0812 02:39:05.100285 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:05.100303 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:05.104752 (+ 4449us) server_negotiation.cc:407] Connection header received
0812 02:39:05.104848 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:05.104854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:05.105110 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:05.105309 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:39:05.106494 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.107502 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.109894 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.110789 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.110899 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:05.111052 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:05.121629 (+ 10577us) server_negotiation.cc:299] Negotiation successful
0812 02:39:05.121878 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":116,"threads_started":1}
I20260812 02:39:05.125720 22434 tablet_service.cc:1467] Processing CreateTablet for tablet d97477afaa5942dfb1213031cfa83b2e (DEFAULT_TABLE table=test-table [id=7a5794c981b04bcf81965a4735505c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:39:05.127128 22434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d97477afaa5942dfb1213031cfa83b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:39:05.129662 22821 tablet_bootstrap.cc:492] T d97477afaa5942dfb1213031cfa83b2e P c8e0a0f12c6248e5af5f3ca69ca4271d: Bootstrap starting.
I20260812 02:39:05.133736 22821 tablet_bootstrap.cc:654] T d97477afaa5942dfb1213031cfa83b2e P c8e0a0f12c6248e5af5f3ca69ca4271d: Neither blocks nor log segments found. Creating new log.
I20260812 02:39:05.135035 22821 log.cc:824] T d97477afaa5942dfb1213031cfa83b2e P c8e0a0f12c6248e5af5f3ca69ca4271d: Log is configured to *not* fsync() on all Append() calls
I20260812 02:39:05.137084 22825 tablet_bootstrap.cc:492] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3: Bootstrap starting.
I20260812 02:39:05.137748 22834 tablet_bootstrap.cc:492] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c: Bootstrap starting.
I20260812 02:39:05.140528 22825 tablet_bootstrap.cc:654] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3: Neither blocks nor log segments found. Creating new log.
I20260812 02:39:05.140657 22834 tablet_bootstrap.cc:654] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c: Neither blocks nor log segments found. Creating new log.
I20260812 02:39:05.141844 22834 log.cc:824] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c: Log is configured to *not* fsync() on all Append() calls
I20260812 02:39:05.141974 22825 log.cc:824] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3: Log is configured to *not* fsync() on all Append() calls
I20260812 02:39:05.144784 22834 tablet_bootstrap.cc:492] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c: No bootstrap required, opened a new log
I20260812 02:39:05.145022 22834 ts_tablet_manager.cc:1397] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 02:39:05.145668 22821 tablet_bootstrap.cc:492] T d97477afaa5942dfb1213031cfa83b2e P c8e0a0f12c6248e5af5f3ca69ca4271d: No bootstrap required, opened a new log
I20260812 02:39:05.145865 22821 ts_tablet_manager.cc:1397] T d97477afaa5942dfb1213031cfa83b2e P c8e0a0f12c6248e5af5f3ca69ca4271d: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260812 02:39:05.150931 22821 raft_consensus.cc:348] T d97477afaa5942dfb1213031cfa83b2e P c8e0a0f12c6248e5af5f3ca69ca4271d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e0a0f12c6248e5af5f3ca69ca4271d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40299 } } peers { permanent_uuid: "239139e741c84a7cbe4302f9e4b827d3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36327 } } peers { permanent_uuid: "8753a9bda42148a8863d664dfee6281c" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 43983 } }
I20260812 02:39:05.150931 22834 raft_consensus.cc:348] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e0a0f12c6248e5af5f3ca69ca4271d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40299 } } peers { permanent_uuid: "239139e741c84a7cbe4302f9e4b827d3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36327 } } peers { permanent_uuid: "8753a9bda42148a8863d664dfee6281c" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 43983 } }
I20260812 02:39:05.151275 22834 raft_consensus.cc:374] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:39:05.151378 22834 raft_consensus.cc:729] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8753a9bda42148a8863d664dfee6281c, State: Initialized, Role: FOLLOWER
I20260812 02:39:05.151427 22821 raft_consensus.cc:374] T d97477afaa5942dfb1213031cfa83b2e P c8e0a0f12c6248e5af5f3ca69ca4271d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:39:05.151500 22821 raft_consensus.cc:729] T d97477afaa5942dfb1213031cfa83b2e P c8e0a0f12c6248e5af5f3ca69ca4271d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8e0a0f12c6248e5af5f3ca69ca4271d, State: Initialized, Role: FOLLOWER
I20260812 02:39:05.152076 22821 consensus_queue.cc:260] T d97477afaa5942dfb1213031cfa83b2e P c8e0a0f12c6248e5af5f3ca69ca4271d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e0a0f12c6248e5af5f3ca69ca4271d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40299 } } peers { permanent_uuid: "239139e741c84a7cbe4302f9e4b827d3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36327 } } peers { permanent_uuid: "8753a9bda42148a8863d664dfee6281c" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 43983 } }
I20260812 02:39:05.152078 22834 consensus_queue.cc:260] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e0a0f12c6248e5af5f3ca69ca4271d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40299 } } peers { permanent_uuid: "239139e741c84a7cbe4302f9e4b827d3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36327 } } peers { permanent_uuid: "8753a9bda42148a8863d664dfee6281c" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 43983 } }
I20260812 02:39:05.153378 22821 ts_tablet_manager.cc:1428] T d97477afaa5942dfb1213031cfa83b2e P c8e0a0f12c6248e5af5f3ca69ca4271d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 02:39:05.153457 22834 ts_tablet_manager.cc:1428] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260812 02:39:05.153844 22345 heartbeater.cc:503] Master 127.3.127.190:37145 was elected leader, sending a full tablet report...
I20260812 02:39:05.153946 22564 heartbeater.cc:503] Master 127.3.127.190:37145 was elected leader, sending a full tablet report...
I20260812 02:39:05.161201 22825 tablet_bootstrap.cc:492] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3: No bootstrap required, opened a new log
I20260812 02:39:05.161545 22825 ts_tablet_manager.cc:1397] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.000s
I20260812 02:39:05.166373 22825 raft_consensus.cc:348] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e0a0f12c6248e5af5f3ca69ca4271d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40299 } } peers { permanent_uuid: "239139e741c84a7cbe4302f9e4b827d3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36327 } } peers { permanent_uuid: "8753a9bda42148a8863d664dfee6281c" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 43983 } }
I20260812 02:39:05.166810 22825 raft_consensus.cc:374] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:39:05.166903 22825 raft_consensus.cc:729] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 239139e741c84a7cbe4302f9e4b827d3, State: Initialized, Role: FOLLOWER
I20260812 02:39:05.167536 22825 consensus_queue.cc:260] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e0a0f12c6248e5af5f3ca69ca4271d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40299 } } peers { permanent_uuid: "239139e741c84a7cbe4302f9e4b827d3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36327 } } peers { permanent_uuid: "8753a9bda42148a8863d664dfee6281c" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 43983 } }
I20260812 02:39:05.168758 22825 ts_tablet_manager.cc:1428] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 02:39:05.169345 22795 heartbeater.cc:503] Master 127.3.127.190:37145 was elected leader, sending a full tablet report...
Aug 12 02:39:05 dist-test-slave-rmhg krb5kdc[21266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502344, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
W20260812 02:39:05.233361 22351 tablet.cc:2369] T d97477afaa5942dfb1213031cfa83b2e P c8e0a0f12c6248e5af5f3ca69ca4271d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:39:05.285198 22796 tablet.cc:2369] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:39:05.366298 22566 tablet.cc:2369] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:39:05.463279 22840 raft_consensus.cc:482] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:39:05.463562 22840 raft_consensus.cc:504] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e0a0f12c6248e5af5f3ca69ca4271d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40299 } } peers { permanent_uuid: "239139e741c84a7cbe4302f9e4b827d3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36327 } } peers { permanent_uuid: "8753a9bda42148a8863d664dfee6281c" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 43983 } }
I20260812 02:39:05.465040 22840 leader_election.cc:288] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8e0a0f12c6248e5af5f3ca69ca4271d (127.3.127.129:40299), 239139e741c84a7cbe4302f9e4b827d3 (127.3.127.131:36327)
I20260812 02:39:05.471597 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:05.466196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:55727 (local address 127.3.127.129:40299)
0812 02:39:05.466406 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:05.466411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:05.466895 (+ 484us) server_negotiation.cc:407] Connection header received
0812 02:39:05.467136 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:05.467140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:05.467195 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:05.467246 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:39:05.467734 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.468458 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.470111 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.470775 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.470880 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:05.470936 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:05.471343 (+ 407us) server_negotiation.cc:299] Negotiation successful
0812 02:39:05.471435 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260812 02:39:05.472466 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:05.465575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:40299 (local address 127.3.127.130:55727)
0812 02:39:05.466789 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:05.466804 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:39:05.466996 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:39:05.467400 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:39:05.467403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:39:05.467422 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:39:05.467578 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:05.467590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.468607 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.468610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:05.469944 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:05.469954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.471032 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.471035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:05.471202 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:05.471229 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 02:39:05.472212 (+ 983us) client_negotiation.cc:240] Negotiation successful
0812 02:39:05.472296 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1087,"mutex_wait_us":23,"thread_start_us":166,"threads_started":1}
I20260812 02:39:05.472594 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:05.466650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:42181 (local address 127.3.127.131:36327)
0812 02:39:05.466776 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:05.466780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:05.468055 (+ 1275us) server_negotiation.cc:407] Connection header received
0812 02:39:05.468243 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:05.468246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:05.468300 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:05.468343 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:39:05.468748 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.469550 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.471342 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.471880 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.471991 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:05.472050 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:05.472400 (+ 350us) server_negotiation.cc:299] Negotiation successful
0812 02:39:05.472485 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 02:39:05.473034 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:05.466247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:36327 (local address 127.3.127.130:42181)
0812 02:39:05.467913 (+ 1666us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:05.467928 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:39:05.468133 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:39:05.468465 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:39:05.468468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:39:05.468484 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:39:05.468623 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:05.468629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.469814 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.469817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:05.471196 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:05.471205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.472158 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.472162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:05.472258 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:05.472278 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 02:39:05.472788 (+ 510us) client_negotiation.cc:240] Negotiation successful
0812 02:39:05.472866 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1594,"mutex_wait_us":39,"thread_start_us":114,"threads_started":1}
I20260812 02:39:05.473654 22284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d97477afaa5942dfb1213031cfa83b2e" candidate_uuid: "8753a9bda42148a8863d664dfee6281c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8e0a0f12c6248e5af5f3ca69ca4271d" is_pre_election: true
I20260812 02:39:05.473847 22697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d97477afaa5942dfb1213031cfa83b2e" candidate_uuid: "8753a9bda42148a8863d664dfee6281c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "239139e741c84a7cbe4302f9e4b827d3" is_pre_election: true
I20260812 02:39:05.474100 22284 raft_consensus.cc:2454] T d97477afaa5942dfb1213031cfa83b2e P c8e0a0f12c6248e5af5f3ca69ca4271d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8753a9bda42148a8863d664dfee6281c in term 0.
I20260812 02:39:05.474176 22697 raft_consensus.cc:2454] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8753a9bda42148a8863d664dfee6281c in term 0.
I20260812 02:39:05.474747 22383 leader_election.cc:302] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8753a9bda42148a8863d664dfee6281c, c8e0a0f12c6248e5af5f3ca69ca4271d; no voters:
I20260812 02:39:05.475101 22840 raft_consensus.cc:2781] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:39:05.475178 22840 raft_consensus.cc:482] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:39:05.475215 22840 raft_consensus.cc:3037] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:39:05.476552 22840 raft_consensus.cc:504] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e0a0f12c6248e5af5f3ca69ca4271d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40299 } } peers { permanent_uuid: "239139e741c84a7cbe4302f9e4b827d3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36327 } } peers { permanent_uuid: "8753a9bda42148a8863d664dfee6281c" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 43983 } }
I20260812 02:39:05.476979 22840 leader_election.cc:288] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c [CANDIDATE]: Term 1 election: Requested vote from peers c8e0a0f12c6248e5af5f3ca69ca4271d (127.3.127.129:40299), 239139e741c84a7cbe4302f9e4b827d3 (127.3.127.131:36327)
I20260812 02:39:05.477427 22284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d97477afaa5942dfb1213031cfa83b2e" candidate_uuid: "8753a9bda42148a8863d664dfee6281c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8e0a0f12c6248e5af5f3ca69ca4271d"
I20260812 02:39:05.477586 22284 raft_consensus.cc:3037] T d97477afaa5942dfb1213031cfa83b2e P c8e0a0f12c6248e5af5f3ca69ca4271d [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:39:05.477687 22697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d97477afaa5942dfb1213031cfa83b2e" candidate_uuid: "8753a9bda42148a8863d664dfee6281c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "239139e741c84a7cbe4302f9e4b827d3"
I20260812 02:39:05.477823 22697 raft_consensus.cc:3037] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:39:05.478876 22284 raft_consensus.cc:2454] T d97477afaa5942dfb1213031cfa83b2e P c8e0a0f12c6248e5af5f3ca69ca4271d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8753a9bda42148a8863d664dfee6281c in term 1.
I20260812 02:39:05.479373 22697 raft_consensus.cc:2454] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8753a9bda42148a8863d664dfee6281c in term 1.
I20260812 02:39:05.479408 22383 leader_election.cc:302] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8753a9bda42148a8863d664dfee6281c, c8e0a0f12c6248e5af5f3ca69ca4271d; no voters:
I20260812 02:39:05.479627 22840 raft_consensus.cc:2781] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:39:05.479868 22840 raft_consensus.cc:686] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c [term 1 LEADER]: Becoming Leader. State: Replica: 8753a9bda42148a8863d664dfee6281c, State: Running, Role: LEADER
I20260812 02:39:05.480289 22840 consensus_queue.cc:237] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e0a0f12c6248e5af5f3ca69ca4271d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40299 } } peers { permanent_uuid: "239139e741c84a7cbe4302f9e4b827d3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36327 } } peers { permanent_uuid: "8753a9bda42148a8863d664dfee6281c" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 43983 } }
I20260812 02:39:05.484824 22000 catalog_manager.cc:5495] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8753a9bda42148a8863d664dfee6281c (127.3.127.130). New cstate: current_term: 1 leader_uuid: "8753a9bda42148a8863d664dfee6281c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e0a0f12c6248e5af5f3ca69ca4271d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "239139e741c84a7cbe4302f9e4b827d3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8753a9bda42148a8863d664dfee6281c" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 43983 } health_report { overall_health: HEALTHY } } }
I20260812 02:39:05.560883 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:05.555998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47266 (local address 127.3.127.130:43983)
0812 02:39:05.556125 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:05.556130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:05.556146 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:39:05.556331 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:05.556334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:05.556410 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:05.556538 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:39:05.556791 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.557623 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.558905 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:05.559241 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:05.559346 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:05.559760 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:39:05.560379 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:39:05.560517 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:05.560592 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 02:39:05.560727 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 02:39:05.569067 22697 raft_consensus.cc:1261] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3 [term 1 FOLLOWER]: Refusing update from remote peer 8753a9bda42148a8863d664dfee6281c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:39:05.569067 22284 raft_consensus.cc:1261] T d97477afaa5942dfb1213031cfa83b2e P c8e0a0f12c6248e5af5f3ca69ca4271d [term 1 FOLLOWER]: Refusing update from remote peer 8753a9bda42148a8863d664dfee6281c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:39:05.570072 22859 consensus_queue.cc:1035] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c [LEADER]: Connected to new peer: Peer: permanent_uuid: "239139e741c84a7cbe4302f9e4b827d3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 36327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:39:05.570353 22840 consensus_queue.cc:1035] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8e0a0f12c6248e5af5f3ca69ca4271d" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:39:05.579862 22868 mvcc.cc:205] Tried to move back new op lower bound from 7317513607441711104 to 7317513607090565120. Current Snapshot: MvccSnapshot[applied={T|T < 7317513607441711104}]
I20260812 02:39:05.583415 22870 mvcc.cc:205] Tried to move back new op lower bound from 7317513607441711104 to 7317513607090565120. Current Snapshot: MvccSnapshot[applied={T|T < 7317513607441711104}]
I20260812 02:39:05.588552 22871 mvcc.cc:205] Tried to move back new op lower bound from 7317513607441711104 to 7317513607090565120. Current Snapshot: MvccSnapshot[applied={T|T < 7317513607441711104}]
I20260812 02:39:05.598361 22006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:39:05.598573 22006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:39:05.601429 22006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84c89d48a0184900bdb27fd2a4611360: Sending DeleteTablet for 3 replicas of tablet d97477afaa5942dfb1213031cfa83b2e
I20260812 02:39:05.602399 22219 tablet_service.cc:1514] Processing DeleteTablet for tablet d97477afaa5942dfb1213031cfa83b2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:39:05 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:60916
I20260812 02:39:05.602607 22678 tablet_service.cc:1514] Processing DeleteTablet for tablet d97477afaa5942dfb1213031cfa83b2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:39:05 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:57408
I20260812 02:39:05.602994 22434 tablet_service.cc:1514] Processing DeleteTablet for tablet d97477afaa5942dfb1213031cfa83b2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:39:05 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:47250
I20260812 02:39:05.603927 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 22141
I20260812 02:39:05.634079 22899 tablet_replica.cc:330] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c: stopping tablet replica
I20260812 02:39:05.635273 22899 raft_consensus.cc:2229] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:39:05.639995 22899 raft_consensus.cc:2258] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:39:05.642251 22899 ts_tablet_manager.cc:1905] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:39:05.642700 22895 tablet_replica.cc:330] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3: stopping tablet replica
I20260812 02:39:05.646384 22899 ts_tablet_manager.cc:1918] T d97477afaa5942dfb1213031cfa83b2e P 8753a9bda42148a8863d664dfee6281c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:39:05.647015 22895 raft_consensus.cc:2229] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:39:05.647501 22895 raft_consensus.cc:2258] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:39:05.649466 22895 ts_tablet_manager.cc:1905] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 02:39:05.649540 21980 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.127.129:40299 (error 108)
I20260812 02:39:05.650504 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 22359
I20260812 02:39:05.660584 22895 ts_tablet_manager.cc:1918] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:39:05.661016 22895 log.cc:1196] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/d97477afaa5942dfb1213031cfa83b2e
I20260812 02:39:05.661528 22895 ts_tablet_manager.cc:1939] T d97477afaa5942dfb1213031cfa83b2e P 239139e741c84a7cbe4302f9e4b827d3: Deleting consensus metadata
I20260812 02:39:05.664194 21981 catalog_manager.cc:4841] TS 239139e741c84a7cbe4302f9e4b827d3 (127.3.127.131:36327): tablet d97477afaa5942dfb1213031cfa83b2e (table test-table [id=7a5794c981b04bcf81965a4735505c28]) successfully deleted
I20260812 02:39:05.664866 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:05.663903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:40299 (local address 127.0.0.1:60924)
0812 02:39:05.664666 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:05.664758 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.129:40299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":77,"threads_started":1}
W20260812 02:39:05.665145 21980 catalog_manager.cc:4584] TS c8e0a0f12c6248e5af5f3ca69ca4271d (127.3.127.129:40299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d97477afaa5942dfb1213031cfa83b2e: Network error: Client connection negotiation failed: client connection to 127.3.127.129:40299: connect: Connection refused (error 111)
I20260812 02:39:05.669553 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:05.669221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:43983 (local address 127.0.0.1:47282)
0812 02:39:05.669408 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:05.669462 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.130:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260812 02:39:05.669627 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 22576
I20260812 02:39:05.690725 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 21947
I20260812 02:39:05.710784 3582 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 02:39:06.059022 3582 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 02:39:06.059175 3582 mini_ranger.cc:66] Stopping Ranger...
I20260812 02:39:06.406872 3582 mini_ranger.cc:68] Stopped Ranger
2026-08-12 02:39:06.407 UTC [21318] LOG: received smart shutdown request
2026-08-12 02:39:06.411 UTC [21318] LOG: background worker "logical replication launcher" (PID 21335) exited with exit code 1
2026-08-12 02:39:06.412 UTC [21327] LOG: shutting down
2026-08-12 02:39:06.600 UTC [21318] LOG: database system is shut down
2026-08-12T02:39:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52684 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:39:06 dist-test-slave-rmhg krb5kdc[22919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:39:06 dist-test-slave-rmhg krb5kdc[22919](info): set up 2 sockets
Aug 12 02:39:06 dist-test-slave-rmhg krb5kdc[22919](info): commencing operation
krb5kdc: starting...
W20260812 02:39:07.892071 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:39:07 dist-test-slave-rmhg krb5kdc[22919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:39:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:39:07Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.127.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.127.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.127.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/rangeradmin_127.3.127.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.127.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/rangeradmin_127.3.127.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.127.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.127.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.127.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/rangerlookup_127.3.127.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.127.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/rangerlookup_127.3.127.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.127.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.127.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.127.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/HTTP_127.3.127.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.127.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/HTTP_127.3.127.148@KRBTEST.COM.keytab.
I20260812 02:39:08.091470 3582 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task9rbVbc/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/postgres -l logfile start
2026-08-12 02:39:10.794 UTC [22969] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 02:39:10.794 UTC [22969] LOG: listening on IPv4 address "127.3.127.148", port 35265
2026-08-12 02:39:10.798 UTC [22969] LOG: listening on Unix socket "/tmp/.s.PGSQL.35265"
2026-08-12 02:39:10.821 UTC [22973] LOG: database system was shut down at 2026-08-12 02:39:09 UTC
2026-08-12 02:39:10.825 UTC [22969] LOG: database system is ready to accept connections
I20260812 02:39:11.912482 3582 mini_postgres.cc:96] Postgres bound to 35265
2026-08-12 02:39:11.919 UTC [22982] FATAL: database "slave" does not exist
127.3.127.148:35265 - accepting connections
I20260812 02:39:11.920493 3582 mini_ranger.cc:161] Starting Ranger...
I20260812 02:39:11.936535 3582 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 02:39:12.468629 3582 mini_ranger.cc:87] Created ranger Postgres database
I20260812 02:39:12.468721 3582 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ranger-admin
2026-08-12 02:39:12,756 [I] DB FLAVOR :POSTGRES
2026-08-12 02:39:12,757 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 02:39:12,757 [I] Checking connection..
2026-08-12 02:39:12,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 02:39:13,036 [I] Checking connection passed.
2026-08-12 02:39:13,037 [I] --------- Verifying version history table ---------
2026-08-12 02:39:13,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 02:39:13,314 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 02:39:13,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 02:39:13,602 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 02:39:13,603 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 02:39:13,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 02:39:13.880 UTC [23082] WARNING: there is no transaction in progress
2026-08-12 02:39:13,889 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 02:39:13,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 02:39:14,169 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 02:39:14,170 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 02:39:14,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 02:39:14,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 02:39:14,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rmhg.c.gcp-upstream.internal','N') ;"
2026-08-12 02:39:14,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 02:39:15,223 [I] Table x_portal_user does not exist in database ranger
2026-08-12 02:39:15,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 02:39:15,495 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 02:39:15,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 02:39:15,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 02:39:16,024 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 02:39:16,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 02:39:17.090 UTC [23256] WARNING: there is no transaction in progress
2026-08-12 02:39:17.106 UTC [23256] WARNING: there is no transaction in progress
2026-08-12 02:39:17.124 UTC [23256] WARNING: there is no transaction in progress
2026-08-12 02:39:17.141 UTC [23256] WARNING: there is no transaction in progress
2026-08-12 02:39:17.159 UTC [23256] WARNING: there is no transaction in progress
2026-08-12 02:39:17.181 UTC [23256] WARNING: there is no transaction in progress
2026-08-12 02:39:17.250 UTC [23256] WARNING: there is no transaction in progress
2026-08-12 02:39:17.258 UTC [23256] WARNING: there is no transaction in progress
2026-08-12 02:39:17.274 UTC [23256] WARNING: there is no transaction in progress
2026-08-12 02:39:17.285 UTC [23256] WARNING: there is no transaction in progress
2026-08-12 02:39:17.298 UTC [23256] WARNING: there is no transaction in progress
2026-08-12 02:39:17.314 UTC [23256] WARNING: there is no transaction in progress
2026-08-12 02:39:17.325 UTC [23256] WARNING: there is no transaction in progress
2026-08-12 02:39:17.333 UTC [23256] WARNING: there is no transaction in progress
2026-08-12 02:39:17.342 UTC [23256] WARNING: there is no transaction in progress
2026-08-12 02:39:17.352 UTC [23256] WARNING: there is no transaction in progress
2026-08-12 02:39:18,108 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 02:39:18,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 02:39:18,367 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 02:39:18,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 02:39:18,648 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 02:39:18,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 02:39:18,917 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 02:39:18,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 02:39:19,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 02:39:19,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rmhg.c.gcp-upstream.internal';"
2026-08-12 02:39:19,741 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 02:39:19,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 02:39:20,006 [I] DB_PATCHES have already been applied
I20260812 02:39:20.012866 3582 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ranger-admin:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task9rbVbc/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 02:39:20.012946 3582 mini_ranger.cc:193] Using host: 127.3.127.148
I20260812 02:39:20.017778 3582 mini_ranger.cc:227] Ranger admin URL: http://127.3.127.148:47759
Aug 12, 2026 2:39:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:39:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:39:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 2:39:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 2:39:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 2:39:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 2:39:20 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 02:39:21.261080 3582 mini_ranger.cc:160] Time spent starting Ranger: real 9.341s user 0.000s sys 0.012s
Aug 12, 2026 2:39:42 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 2:39:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 2:39:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 2:39:42 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 2:39:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 2:39:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 2:39:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 2:39:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 2:39:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 2:39:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 2:39:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 2:39:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 2:39:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 2:39:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 2:39:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 2:39:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 2:39:42 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 2:39:43 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 02:39:44.486240 3582 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.127.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.127.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.127.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/rangerkms_127.3.127.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.127.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/rangerkms_127.3.127.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.127.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.127.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.127.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/HTTP_127.3.127.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.127.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/HTTP_127.3.127.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 02:39:44 dist-test-slave-rmhg krb5kdc[22919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502384, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 02:39:44.589493 3582 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 02:39:44.601548 3582 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 02:39:45.467079 3582 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 02:39:45.467160 3582 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ranger-kms
I20260812 02:39:45.467172 3582 mini_ranger_kms.cc:227] Using postgres at 127.3.127.148:35265
2026-08-12 02:39:45,627 [I] DB FLAVOR :POSTGRES
2026-08-12 02:39:45,628 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 02:39:45,628 [I] Checking connection
2026-08-12 02:39:45,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 02:39:45,883 [I] connection success
2026-08-12 02:39:45,884 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 02:39:45,884 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 02:39:45,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 02:39:46,147 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 02:39:46,148 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 02:39:46,148 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 02:39:46,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.127.148:35265/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 02:39:46,428 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 02:39:46.980302 3582 mini_ranger_kms.cc:310] Created kms service
I20260812 02:39:47.218549 3582 mini_ranger_kms.cc:326] Created kudu user
I20260812 02:39:47.272979 3582 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 02:39:47 dist-test-slave-rmhg krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502384, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
I20260812 02:39:47.735569 3582 mini_ranger_kms.cc:384] Added ranger policy
I20260812 02:39:47.735846 3582 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ranger-kms:/tmp/dist-test-task9rbVbc/build/debug/bin/postgresql.jar:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task9rbVbc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 02:39:47.735874 3582 mini_ranger_kms.cc:240] Using host: 127.3.127.148
I20260812 02:39:47.741719 3582 mini_ranger_kms.cc:276] Ranger KMS PID: 23540
I20260812 02:39:47.741802 3582 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.127.148:52931
Aug 12, 2026 2:39:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:39:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:39:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 2:39:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 2:39:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 2:39:49 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 02:39:50 dist-test-slave-rmhg krb5kdc[22919](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786502390, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.127.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:39:50.352759 3582 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.763s user 0.010s sys 0.136s
I20260812 02:39:50.352902 3582 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 02:39:50.352995 3582 mini_ranger_kms.cc:401] 127.3.127.148:52931/kms/v1/keys
Aug 12 02:39:50 dist-test-slave-rmhg krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502390, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.127.148@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
Aug 12 02:39:51 dist-test-slave-rmhg krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502390, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.127.148@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
Aug 12 02:39:53 dist-test-slave-rmhg krb5kdc[22919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:39:53.620214 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:45045
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:45045
--ranger_config_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.127.148:52931/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:39:53.740067 23632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:39:53.740357 23632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:39:53.740397 23632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:39:53.744693 23632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:39:53.744776 23632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:39:53.744796 23632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:39:53.744810 23632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:39:53.744824 23632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:39:53.750710 23632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.127.148:52931/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:45045
--ranger_config_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:45045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:39:53.751859 23632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:39:53.752867 23632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:39:53.761821 23638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:39:53.761895 23632 server_base.cc:1034] running on GCE node
W20260812 02:39:53.761821 23648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:39:53.761899 23641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:39:53.762785 23632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:39:53.764014 23632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:39:53.765224 23632 hybrid_clock.cc:648] HybridClock initialized: now 1786502393765206 us; error 32 us; skew 500 ppm
Aug 12 02:39:53 dist-test-slave-rmhg krb5kdc[22919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502393, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:39:53.768313 23632 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:39:53.769415 23632 webserver.cc:460] Webserver started at http://127.3.127.190:38683/ using document root <none> and password file <none>
I20260812 02:39:53.770027 23632 fs_manager.cc:359] Metadata directory not provided
I20260812 02:39:53.770090 23632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:39:53.770299 23632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:39:53 dist-test-slave-rmhg krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502393, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
I20260812 02:39:53.839942 23632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "5748ff60a4dc4112a4de838288d6dc81"
format_stamp: "Formatted at 2026-08-12 02:39:53 on dist-test-slave-rmhg"
server_key: "a24535c2681bcce9eb73c651aaf16e04"
server_key_iv: "e0ff10597c9854c6a53933c0bfc0a73f"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:53.840597 23632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "5748ff60a4dc4112a4de838288d6dc81"
format_stamp: "Formatted at 2026-08-12 02:39:53 on dist-test-slave-rmhg"
server_key: "a24535c2681bcce9eb73c651aaf16e04"
server_key_iv: "e0ff10597c9854c6a53933c0bfc0a73f"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:53.844957 23632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 02:39:53.865101 23660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:53.866988 23632 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:39:53.867169 23632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "5748ff60a4dc4112a4de838288d6dc81"
format_stamp: "Formatted at 2026-08-12 02:39:53 on dist-test-slave-rmhg"
server_key: "a24535c2681bcce9eb73c651aaf16e04"
server_key_iv: "e0ff10597c9854c6a53933c0bfc0a73f"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:53.867321 23632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:39:53.891973 23632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:39:53.895828 23632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:39:53.896060 23632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:39:53.905306 23632 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:45045
I20260812 02:39:53.905306 23745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:45045 every 8 connection(s)
I20260812 02:39:53.906694 23632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:39:53.909929 23747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:39:53.910347 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 23632
I20260812 02:39:53.910482 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:39:53.910831 3582 external_mini_cluster.cc:1428] Setting key 886f1fe84231e6c3c159ec7b80db442e
I20260812 02:39:53.917140 23747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81: Bootstrap starting.
I20260812 02:39:53.919749 23747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81: Neither blocks nor log segments found. Creating new log.
I20260812 02:39:53.920624 23747 log.cc:824] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:39:53 dist-test-slave-rmhg krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:39:53.922812 23747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81: No bootstrap required, opened a new log
I20260812 02:39:53.926932 23747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5748ff60a4dc4112a4de838288d6dc81" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 45045 } }
I20260812 02:39:53.927241 23747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:39:53.927306 23747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5748ff60a4dc4112a4de838288d6dc81, State: Initialized, Role: FOLLOWER
I20260812 02:39:53.927855 23747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5748ff60a4dc4112a4de838288d6dc81" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 45045 } }
I20260812 02:39:53.928063 23747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:39:53.928146 23747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:39:53.928258 23747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:39:53.929018 23753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:53.912502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58062 (local address 127.3.127.190:45045)
0812 02:39:53.913037 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:53.913057 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:53.913106 (+ 49us) server_negotiation.cc:407] Connection header received
0812 02:39:53.913957 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:53.913985 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:53.914440 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:53.914819 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:39:53.915772 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:53.917285 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:53.918575 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:53.919281 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:53.919438 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:53.922667 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:39:53.922673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:39:53.922692 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:39:53.922724 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:39:53.925617 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:53.926212 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:53.926219 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:53.926225 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:53.926374 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:53.926714 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:53.926717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:53.926719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:53.927287 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:39:53.927502 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:53.927853 (+ 351us) server_negotiation.cc:299] Negotiation successful
0812 02:39:53.928181 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":160,"threads_started":1}
I20260812 02:39:53.929286 23747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5748ff60a4dc4112a4de838288d6dc81" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 45045 } }
I20260812 02:39:53.929641 23747 leader_election.cc:302] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5748ff60a4dc4112a4de838288d6dc81; no voters:
I20260812 02:39:53.929960 23747 leader_election.cc:288] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:39:53.930073 23756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:39:53.931241 23756 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [term 1 LEADER]: Becoming Leader. State: Replica: 5748ff60a4dc4112a4de838288d6dc81, State: Running, Role: LEADER
I20260812 02:39:53.931317 23747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:39:53.931592 23756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5748ff60a4dc4112a4de838288d6dc81" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 45045 } }
I20260812 02:39:53.933411 23758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5748ff60a4dc4112a4de838288d6dc81. Latest consensus state: current_term: 1 leader_uuid: "5748ff60a4dc4112a4de838288d6dc81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5748ff60a4dc4112a4de838288d6dc81" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 45045 } } }
I20260812 02:39:53.933370 23757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5748ff60a4dc4112a4de838288d6dc81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5748ff60a4dc4112a4de838288d6dc81" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 45045 } } }
I20260812 02:39:53.933549 23758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [sys.catalog]: This master's current role is: LEADER
I20260812 02:39:53.933578 23757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [sys.catalog]: This master's current role is: LEADER
I20260812 02:39:53.933616 23747 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 02:39:53.934653 23780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:39:53.940321 23780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:39:53.946730 23780 catalog_manager.cc:1347] Generated new cluster ID: 9b64c8ce713048ddb49e87e88abe5892
I20260812 02:39:53.946812 23780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:39:53.964509 23780 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:39:53.965669 23780 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:39:53.976579 23780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81: Generated new TSK 0
I20260812 02:39:53.977591 23780 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
I20260812 02:39:54.687989 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.127.148:52931/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:45045
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:39:54.816342 23819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:39:54.816671 23819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:39:54.816735 23819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:39:54.821244 23819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:39:54.821326 23819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:39:54.821411 23819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:39:54.827850 23819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.127.148:52931/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:45045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:39:54.829182 23819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:39:54.830675 23819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:39:54.843750 23828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:39:54.843825 23831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:39:54.843822 23829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:39:54.844269 23819 server_base.cc:1034] running on GCE node
I20260812 02:39:54.844807 23819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:39:54.845700 23819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:39:54.846911 23819 hybrid_clock.cc:648] HybridClock initialized: now 1786502394846865 us; error 71 us; skew 500 ppm
Aug 12 02:39:54 dist-test-slave-rmhg krb5kdc[22919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502394, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:39:54.849923 23819 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:39:54.851001 23819 webserver.cc:460] Webserver started at http://127.3.127.129:33277/ using document root <none> and password file <none>
I20260812 02:39:54.851600 23819 fs_manager.cc:359] Metadata directory not provided
I20260812 02:39:54.851673 23819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:39:54.851948 23819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:39:54 dist-test-slave-rmhg krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502394, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
I20260812 02:39:54.866526 23819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "e12deda4f6e74e8ca67fced1490efd69"
format_stamp: "Formatted at 2026-08-12 02:39:54 on dist-test-slave-rmhg"
server_key: "af26c82cbd29f337fcc16c9152b48d91"
server_key_iv: "2a61500b16a1c2dc6fdd605ae554d3ed"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:54.867479 23819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "e12deda4f6e74e8ca67fced1490efd69"
format_stamp: "Formatted at 2026-08-12 02:39:54 on dist-test-slave-rmhg"
server_key: "af26c82cbd29f337fcc16c9152b48d91"
server_key_iv: "2a61500b16a1c2dc6fdd605ae554d3ed"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:54.874119 23819 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260812 02:39:54.892278 23842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:54.894435 23819 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260812 02:39:54.894631 23819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "e12deda4f6e74e8ca67fced1490efd69"
format_stamp: "Formatted at 2026-08-12 02:39:54 on dist-test-slave-rmhg"
server_key: "af26c82cbd29f337fcc16c9152b48d91"
server_key_iv: "2a61500b16a1c2dc6fdd605ae554d3ed"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:54.894815 23819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:39:54.915390 23819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:39:54.919303 23819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:39:54.919597 23819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:39:54.920440 23819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:39:54.921684 23819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:39:54.921801 23819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:54.921937 23819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:39:54.922021 23819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:54.933030 23819 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:33727
I20260812 02:39:54.934382 23819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:39:54.942745 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 23819
I20260812 02:39:54.942896 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:39:54.943241 3582 external_mini_cluster.cc:1428] Setting key 850ce2069703d91dd6eb46bb789ea7bb
WARNING: no policy specified for kudu/127.3.127.130@KRBTEST.COM; defaulting to no policy
I20260812 02:39:54.959585 24050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:33727 every 8 connection(s)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.130@KRBTEST.COM" created.
Aug 12 02:39:54 dist-test-slave-rmhg krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502394, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:39:54.979604 24058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:54.944180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:46509 (local address 127.3.127.190:45045)
0812 02:39:54.945116 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:54.945127 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:54.957926 (+ 12799us) server_negotiation.cc:407] Connection header received
0812 02:39:54.959389 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:54.959393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:54.959469 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:54.959595 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:39:54.964189 (+ 4594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:54.965190 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:54.967225 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:54.967613 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:54.967707 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:54.973118 (+ 5411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:39:54.973123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:39:54.973126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:39:54.973214 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:39:54.975413 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:54.976373 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:54.976379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:54.976381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:54.976492 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:54.977917 (+ 1425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:54.977922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:54.977924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:54.978210 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:39:54.978309 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:54.979221 (+ 912us) server_negotiation.cc:299] Negotiation successful
0812 02:39:54.979387 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":114,"threads_started":1}
I20260812 02:39:54.981668 24055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:54.942872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:45045 (local address 127.3.127.129:46509)
0812 02:39:54.957654 (+ 14782us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:54.957758 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0812 02:39:54.959009 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:39:54.960138 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:39:54.960162 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:39:54.960622 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:39:54.961497 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:54.961528 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:54.965424 (+ 3896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:54.965428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:54.967047 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:54.967065 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:54.967983 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:54.967988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:54.968246 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:54.969034 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:39:54.969059 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:39:54.971277 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:39:54.975690 (+ 4413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:39:54.975699 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:39:54.975723 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:39:54.976236 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:39:54.977337 (+ 1101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:39:54.977341 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:39:54.977343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:39:54.977615 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:39:54.978389 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:39:54.978397 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:39:54.978874 (+ 477us) client_negotiation.cc:769] Sending connection context
0812 02:39:54.979896 (+ 1022us) client_negotiation.cc:240] Negotiation successful
0812 02:39:54.980361 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14406,"thread_start_us":164,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
I20260812 02:39:54.983614 24051 heartbeater.cc:347] Connected to a master server at 127.3.127.190:45045
I20260812 02:39:54.983901 24051 heartbeater.cc:464] Registering TS with master...
I20260812 02:39:54.984328 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.127.148:52931/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:45045
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:39:54.984633 24051 heartbeater.cc:511] Master 127.3.127.190:45045 requested a full tablet report, sending...
I20260812 02:39:54.987025 23687 ts_manager.cc:194] Registered new tserver with Master: e12deda4f6e74e8ca67fced1490efd69 (127.3.127.129:33727)
I20260812 02:39:54.988744 23687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:46509
Aug 12 02:39:55 dist-test-slave-rmhg krb5kdc[22919](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786502395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 02:39:55.158349 24065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:39:55.158871 24065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:39:55.158967 24065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:39:55.165431 24065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:39:55.165575 24065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:39:55.165685 24065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:39:55.174233 24065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.127.148:52931/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:45045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:39:55.176023 24065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:39:55.177646 24065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:39:55.192214 24073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:39:55.192334 24074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:39:55.192227 24076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:39:55.193885 24065 server_base.cc:1034] running on GCE node
I20260812 02:39:55.194518 24065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:39:55.195464 24065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:39:55.196751 24065 hybrid_clock.cc:648] HybridClock initialized: now 1786502395196721 us; error 47 us; skew 500 ppm
Aug 12 02:39:55 dist-test-slave-rmhg krb5kdc[22919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:39:55.201797 24065 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:39:55.203419 24065 webserver.cc:460] Webserver started at http://127.3.127.130:42805/ using document root <none> and password file <none>
I20260812 02:39:55.204324 24065 fs_manager.cc:359] Metadata directory not provided
I20260812 02:39:55.204525 24065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:39:55.204895 24065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:39:55 dist-test-slave-rmhg krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
I20260812 02:39:55.231798 24065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "63099c0e60854c838ea93538c1e33ad0"
format_stamp: "Formatted at 2026-08-12 02:39:55 on dist-test-slave-rmhg"
server_key: "a15953273f1f200ac4bee58c955e0cd2"
server_key_iv: "6fef3809eef3384d0e5639a068e7b0b6"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:55.232765 24065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "63099c0e60854c838ea93538c1e33ad0"
format_stamp: "Formatted at 2026-08-12 02:39:55 on dist-test-slave-rmhg"
server_key: "a15953273f1f200ac4bee58c955e0cd2"
server_key_iv: "6fef3809eef3384d0e5639a068e7b0b6"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:55.239507 24065 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260812 02:39:55.262598 24088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:55.264820 24065 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260812 02:39:55.264992 24065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "63099c0e60854c838ea93538c1e33ad0"
format_stamp: "Formatted at 2026-08-12 02:39:55 on dist-test-slave-rmhg"
server_key: "a15953273f1f200ac4bee58c955e0cd2"
server_key_iv: "6fef3809eef3384d0e5639a068e7b0b6"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:55.265196 24065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:39:55.292878 24065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:39:55.297021 24065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:39:55.297299 24065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:39:55.298225 24065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:39:55.299433 24065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:39:55.299506 24065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:55.299568 24065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:39:55.299597 24065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:55.311954 24065 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:35575
I20260812 02:39:55.313400 24065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:39:55.314937 24296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:35575 every 8 connection(s)
I20260812 02:39:55.321249 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 24065
I20260812 02:39:55.321394 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:39:55.321755 3582 external_mini_cluster.cc:1428] Setting key 8b73790d15350a20ee94cfa6bf7426f8
Aug 12 02:39:55 dist-test-slave-rmhg krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.127.131@KRBTEST.COM; defaulting to no policy
I20260812 02:39:55.339372 24058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:55.318479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:39325 (local address 127.3.127.190:45045)
0812 02:39:55.318638 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:55.318643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:55.318665 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:39:55.319405 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:55.319409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:55.319467 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:55.319521 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:39:55.323706 (+ 4185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:55.324486 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:55.326417 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:55.326800 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:55.326932 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:55.334162 (+ 7230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:39:55.334168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:39:55.334171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:39:55.334213 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:39:55.336144 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:55.337216 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:55.337221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:55.337223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:55.337329 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:55.337871 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:55.337875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:55.337877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:55.338142 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:39:55.338262 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:55.339018 (+ 756us) server_negotiation.cc:299] Negotiation successful
0812 02:39:55.339179 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.131@KRBTEST.COM" created.
I20260812 02:39:55.341382 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:55.317326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:45045 (local address 127.3.127.130:39325)
0812 02:39:55.317948 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:55.318005 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 02:39:55.319129 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:39:55.319957 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:39:55.319967 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:39:55.320514 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:39:55.323516 (+ 3002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:55.323542 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:55.324688 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:55.324691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:55.326247 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:55.326261 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:55.327160 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:55.327165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:55.327395 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:55.329529 (+ 2134us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:39:55.329557 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:39:55.331683 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:39:55.336455 (+ 4772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:39:55.336474 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:39:55.336500 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:39:55.337033 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:39:55.337464 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:39:55.337468 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:39:55.337470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:39:55.337718 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:39:55.338270 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:39:55.338279 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:39:55.338677 (+ 398us) client_negotiation.cc:769] Sending connection context
0812 02:39:55.338926 (+ 249us) client_negotiation.cc:240] Negotiation successful
0812 02:39:55.339522 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":150,"threads_started":1}
I20260812 02:39:55.342954 24298 heartbeater.cc:347] Connected to a master server at 127.3.127.190:45045
I20260812 02:39:55.343258 24298 heartbeater.cc:464] Registering TS with master...
I20260812 02:39:55.344003 24298 heartbeater.cc:511] Master 127.3.127.190:45045 requested a full tablet report, sending...
I20260812 02:39:55.345387 23687 ts_manager.cc:194] Registered new tserver with Master: 63099c0e60854c838ea93538c1e33ad0 (127.3.127.130:35575)
I20260812 02:39:55.346215 23687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:39325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
I20260812 02:39:55.360322 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.127.148:52931/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:45045
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:39:55.557756 24304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:39:55.558225 24304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:39:55.558353 24304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:39:55.564824 24304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:39:55.565068 24304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:39:55.565237 24304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:39:55.573959 24304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.127.148:52931/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:45045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:39:55.575946 24304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:39:55.577742 24304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:39:55.593923 24314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:39:55.593957 24312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:39:55.594445 24317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:39:55.596009 24304 server_base.cc:1034] running on GCE node
I20260812 02:39:55.596743 24304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:39:55.597684 24304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:39:55.600991 24304 hybrid_clock.cc:648] HybridClock initialized: now 1786502395600898 us; error 99 us; skew 500 ppm
Aug 12 02:39:55 dist-test-slave-rmhg krb5kdc[22919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:39:55.605772 24304 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:39:55.607144 24304 webserver.cc:460] Webserver started at http://127.3.127.131:42205/ using document root <none> and password file <none>
I20260812 02:39:55.607960 24304 fs_manager.cc:359] Metadata directory not provided
I20260812 02:39:55.608117 24304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:39:55.608434 24304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:39:55 dist-test-slave-rmhg krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
I20260812 02:39:55.629419 24304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "c701c28f53d7412f9c787b7d556e0874"
format_stamp: "Formatted at 2026-08-12 02:39:55 on dist-test-slave-rmhg"
server_key: "7120b330a2a91a1310a9355b383d6a7f"
server_key_iv: "3b8c54bffd2a839a79f16358e3702d21"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:55.630403 24304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "c701c28f53d7412f9c787b7d556e0874"
format_stamp: "Formatted at 2026-08-12 02:39:55 on dist-test-slave-rmhg"
server_key: "7120b330a2a91a1310a9355b383d6a7f"
server_key_iv: "3b8c54bffd2a839a79f16358e3702d21"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:55.638361 24304 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
Aug 12 02:39:55 dist-test-slave-rmhg krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
I20260812 02:39:55.665685 24331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:55.667546 24304 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260812 02:39:55.667716 24304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "c701c28f53d7412f9c787b7d556e0874"
format_stamp: "Formatted at 2026-08-12 02:39:55 on dist-test-slave-rmhg"
server_key: "7120b330a2a91a1310a9355b383d6a7f"
server_key_iv: "3b8c54bffd2a839a79f16358e3702d21"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:55.667881 24304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:39:55.702708 24304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:39:55.707185 24304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:39:55.707547 24304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:39:55.708511 24304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:39:55.709717 24304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:39:55.709828 24304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:55.709931 24304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:39:55.709996 24304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:55.725636 24304 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:41887
I20260812 02:39:55.727303 24304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:39:55.737584 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 24304
I20260812 02:39:55.737725 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:39:55.738067 3582 external_mini_cluster.cc:1428] Setting key 5b0a991a888330393a831f7112174055
I20260812 02:39:55.743124 24524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:41887 every 8 connection(s)
Aug 12 02:39:55 dist-test-slave-rmhg krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:39:55.762005 24058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:55.744612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:39519 (local address 127.3.127.190:45045)
0812 02:39:55.744757 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:55.744761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:55.746051 (+ 1290us) server_negotiation.cc:407] Connection header received
0812 02:39:55.747616 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:55.747621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:55.747690 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:55.747748 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:39:55.750058 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:55.750911 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:55.752964 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:55.753379 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:55.753475 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:55.757212 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:39:55.757217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:39:55.757220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:39:55.757255 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:39:55.759085 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:55.759850 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:55.759855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:55.759857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:55.759956 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:55.760453 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:55.760457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:55.760459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:55.760721 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:39:55.760808 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:55.761633 (+ 825us) server_negotiation.cc:299] Negotiation successful
0812 02:39:55.761796 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 02:39:55.764261 24527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:55.744175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:45045 (local address 127.3.127.131:39519)
0812 02:39:55.745834 (+ 1659us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:55.745907 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 02:39:55.747289 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:39:55.748285 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:39:55.748298 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:39:55.748807 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:39:55.749813 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:55.749838 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:55.751119 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:55.751123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:55.752808 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:55.752822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:55.753729 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:55.753734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:55.753963 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:55.754711 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:39:55.754734 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:39:55.756941 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:39:55.759256 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:39:55.759264 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:39:55.759287 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:39:55.759714 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:39:55.760098 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:39:55.760102 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:39:55.760104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:39:55.760328 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:39:55.760903 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:39:55.760910 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:39:55.761316 (+ 406us) client_negotiation.cc:769] Sending connection context
0812 02:39:55.762559 (+ 1243us) client_negotiation.cc:240] Negotiation successful
0812 02:39:55.763004 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1369,"thread_start_us":289,"threads_started":1}
I20260812 02:39:55.765878 24525 heartbeater.cc:347] Connected to a master server at 127.3.127.190:45045
I20260812 02:39:55.766170 24525 heartbeater.cc:464] Registering TS with master...
I20260812 02:39:55.766830 24525 heartbeater.cc:511] Master 127.3.127.190:45045 requested a full tablet report, sending...
I20260812 02:39:55.768092 23687 ts_manager.cc:194] Registered new tserver with Master: c701c28f53d7412f9c787b7d556e0874 (127.3.127.131:41887)
I20260812 02:39:55.768863 23687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:39519
I20260812 02:39:55.779824 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:39:55.780758 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 23819
I20260812 02:39:55.841627 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 24065
I20260812 02:39:55.857645 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 24304
I20260812 02:39:55.885752 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 23632
I20260812 02:39:55.907819 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:45045
--webserver_interface=127.3.127.190
--webserver_port=38683
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:45045
--ranger_config_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.127.148:52931/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.127.148:52931/kms with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:39:56.025909 24529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:39:56.026232 24529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:39:56.026273 24529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:39:56.030341 24529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:39:56.030412 24529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:39:56.030431 24529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:39:56.030445 24529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:39:56.030458 24529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:39:56.036305 24529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.127.148:52931/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:45045
--ranger_config_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:45045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=38683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:39:56.037490 24529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:39:56.038524 24529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:39:56.047528 24536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:39:56.047530 24535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:39:56.047731 24539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:39:56.047850 24529 server_base.cc:1034] running on GCE node
I20260812 02:39:56.048506 24529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:39:56.049763 24529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:39:56.050946 24529 hybrid_clock.cc:648] HybridClock initialized: now 1786502396050926 us; error 35 us; skew 500 ppm
Aug 12 02:39:56 dist-test-slave-rmhg krb5kdc[22919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:39:56.053920 24529 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:39:56.054953 24529 webserver.cc:460] Webserver started at http://127.3.127.190:38683/ using document root <none> and password file <none>
I20260812 02:39:56.055570 24529 fs_manager.cc:359] Metadata directory not provided
I20260812 02:39:56.055655 24529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 02:39:56 dist-test-slave-rmhg krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
I20260812 02:39:56.133214 24529 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:39:56.135842 24549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:56.137655 24529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:39:56.137813 24529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "5748ff60a4dc4112a4de838288d6dc81"
format_stamp: "Formatted at 2026-08-12 02:39:53 on dist-test-slave-rmhg"
server_key: "a24535c2681bcce9eb73c651aaf16e04"
server_key_iv: "e0ff10597c9854c6a53933c0bfc0a73f"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:56.138434 24529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:39:56.151376 24529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:39:56.154170 24529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:39:56.154366 24529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:39:56.161890 24529 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:45045
I20260812 02:39:56.161931 24633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:45045 every 8 connection(s)
I20260812 02:39:56.163096 24529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:39:56.167305 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 24529
I20260812 02:39:56.167821 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.127.148:52931/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:33727
--local_ip_for_outbound_sockets=127.3.127.129
--tserver_master_addrs=127.3.127.190:45045
--webserver_port=33277
--webserver_interface=127.3.127.129
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.127.148:52931/kms with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:39:56.167935 24640 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:39:56.170488 24640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81: Bootstrap starting.
I20260812 02:39:56.188272 24640 log.cc:824] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81: Log is configured to *not* fsync() on all Append() calls
I20260812 02:39:56.196609 24640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:39:56.197058 24640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81: Bootstrap complete.
I20260812 02:39:56.201377 24640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5748ff60a4dc4112a4de838288d6dc81" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 45045 } }
I20260812 02:39:56.201849 24640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5748ff60a4dc4112a4de838288d6dc81, State: Initialized, Role: FOLLOWER
I20260812 02:39:56.202417 24640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5748ff60a4dc4112a4de838288d6dc81" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 45045 } }
I20260812 02:39:56.202562 24640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:39:56.202639 24640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:39:56.202755 24640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [term 1 FOLLOWER]: Advancing to term 2
I20260812 02:39:56.204871 24640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5748ff60a4dc4112a4de838288d6dc81" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 45045 } }
I20260812 02:39:56.205281 24640 leader_election.cc:302] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5748ff60a4dc4112a4de838288d6dc81; no voters:
I20260812 02:39:56.205642 24640 leader_election.cc:288] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 02:39:56.205772 24644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 02:39:56.205991 24644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [term 2 LEADER]: Becoming Leader. State: Replica: 5748ff60a4dc4112a4de838288d6dc81, State: Running, Role: LEADER
I20260812 02:39:56.206337 24644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5748ff60a4dc4112a4de838288d6dc81" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 45045 } }
I20260812 02:39:56.207063 24640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:39:56.207783 24647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5748ff60a4dc4112a4de838288d6dc81. Latest consensus state: current_term: 2 leader_uuid: "5748ff60a4dc4112a4de838288d6dc81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5748ff60a4dc4112a4de838288d6dc81" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 45045 } } }
I20260812 02:39:56.207924 24647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [sys.catalog]: This master's current role is: LEADER
I20260812 02:39:56.208247 24645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5748ff60a4dc4112a4de838288d6dc81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5748ff60a4dc4112a4de838288d6dc81" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 45045 } } }
I20260812 02:39:56.208335 24645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81 [sys.catalog]: This master's current role is: LEADER
I20260812 02:39:56.208834 24655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:39:56.212183 24655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:39:56.212826 24640 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 02:39:56.212901 24655 catalog_manager.cc:1259] Loaded cluster ID: 9b64c8ce713048ddb49e87e88abe5892
I20260812 02:39:56.212960 24655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:39:56.215524 24655 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:39:56.216328 24655 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81: Loaded TSK: 0
I20260812 02:39:56.217730 24655 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 02:39:56.318565 24642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:39:56.318959 24642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:39:56.319025 24642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:39:56.325292 24642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:39:56.325388 24642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:39:56.325476 24642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:39:56.333377 24642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.127.148:52931/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:33727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=33277
--tserver_master_addrs=127.3.127.190:45045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:39:56.334800 24642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:39:56.336205 24642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:39:56.349949 24697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:39:56.349949 24700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:39:56.350294 24702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:39:56.350850 24642 server_base.cc:1034] running on GCE node
I20260812 02:39:56.351423 24642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:39:56.352357 24642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:39:56.359790 24642 hybrid_clock.cc:648] HybridClock initialized: now 1786502396359728 us; error 58 us; skew 500 ppm
Aug 12 02:39:56 dist-test-slave-rmhg krb5kdc[22919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:39:56.368026 24642 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:39:56.369535 24642 webserver.cc:460] Webserver started at http://127.3.127.129:33277/ using document root <none> and password file <none>
I20260812 02:39:56.370342 24642 fs_manager.cc:359] Metadata directory not provided
I20260812 02:39:56.370458 24642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 02:39:56 dist-test-slave-rmhg krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
I20260812 02:39:56.419929 24642 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.003s sys 0.001s
I20260812 02:39:56.426360 24713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:56.428256 24642 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260812 02:39:56.428426 24642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "e12deda4f6e74e8ca67fced1490efd69"
format_stamp: "Formatted at 2026-08-12 02:39:54 on dist-test-slave-rmhg"
server_key: "af26c82cbd29f337fcc16c9152b48d91"
server_key_iv: "2a61500b16a1c2dc6fdd605ae554d3ed"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:56.429296 24642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:39:56.454196 24642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:39:56.458319 24642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:39:56.458555 24642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:39:56.459362 24642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:39:56.460521 24642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:39:56.460589 24642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:56.460652 24642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:39:56.460682 24642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:56.472720 24642 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:33727
I20260812 02:39:56.472728 24901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:33727 every 8 connection(s)
I20260812 02:39:56.476722 24642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:39:56.484939 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 24642
I20260812 02:39:56.485538 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.127.148:52931/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:35575
--local_ip_for_outbound_sockets=127.3.127.130
--tserver_master_addrs=127.3.127.190:45045
--webserver_port=42805
--webserver_interface=127.3.127.130
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.127.148:52931/kms with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:39:56 dist-test-slave-rmhg krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:39:56.497501 24906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:56.479412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:45045 (local address 127.3.127.129:58705)
0812 02:39:56.479953 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:56.480018 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 02:39:56.481065 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:39:56.482807 (+ 1742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:39:56.482817 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:39:56.483234 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:39:56.483930 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:56.483955 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:56.486106 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:56.486110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:56.487595 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:56.487609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:56.488628 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:56.488633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:56.488825 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:56.489476 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:39:56.489494 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:39:56.491079 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:39:56.494039 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:39:56.494046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:39:56.494064 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:39:56.494384 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:39:56.494760 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:39:56.494763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:39:56.494764 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:39:56.494910 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:39:56.495552 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:39:56.495559 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:39:56.495824 (+ 265us) client_negotiation.cc:769] Sending connection context
0812 02:39:56.496013 (+ 189us) client_negotiation.cc:240] Negotiation successful
0812 02:39:56.496383 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":137,"threads_started":1}
I20260812 02:39:56.497499 24907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:56.479380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:58705 (local address 127.3.127.190:45045)
0812 02:39:56.480077 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:56.480101 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:56.480168 (+ 67us) server_negotiation.cc:407] Connection header received
0812 02:39:56.481698 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:56.481718 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:56.482260 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:56.482678 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:39:56.484172 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:56.485923 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:56.487746 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:56.488386 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:56.488626 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:56.491303 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:39:56.491310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:39:56.491330 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:39:56.491367 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:39:56.493856 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:56.494534 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:56.494539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:56.494544 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:56.494660 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:56.495012 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:56.495015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:56.495017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:56.495428 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:39:56.495572 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:56.496146 (+ 574us) server_negotiation.cc:299] Negotiation successful
0812 02:39:56.496414 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":134,"threads_started":1}
I20260812 02:39:56.500897 24902 heartbeater.cc:347] Connected to a master server at 127.3.127.190:45045
I20260812 02:39:56.501215 24902 heartbeater.cc:464] Registering TS with master...
I20260812 02:39:56.501920 24902 heartbeater.cc:511] Master 127.3.127.190:45045 requested a full tablet report, sending...
W20260812 02:39:56.504612 24902 heartbeater.cc:644] Failed to heartbeat to 127.3.127.190:45045 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 02:39:56.504771 24902 heartbeater.cc:464] Registering TS with master...
I20260812 02:39:56.504998 24902 heartbeater.cc:511] Master 127.3.127.190:45045 requested a full tablet report, sending...
I20260812 02:39:56.505930 24902 heartbeater.cc:464] Registering TS with master...
I20260812 02:39:56.506141 24902 heartbeater.cc:511] Master 127.3.127.190:45045 requested a full tablet report, sending...
W20260812 02:39:56.506950 24902 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 02:39:56.625735 24909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:39:56.626035 24909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:39:56.626098 24909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:39:56.631502 24909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:39:56.631610 24909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:39:56.631731 24909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:39:56.640049 24909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.127.148:52931/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:35575
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=42805
--tserver_master_addrs=127.3.127.190:45045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:39:56.641373 24909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:39:56.643033 24909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:39:56.655980 24915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:39:56.655985 24920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:39:56.656468 24909 server_base.cc:1034] running on GCE node
W20260812 02:39:56.656618 24914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:39:56.657063 24909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:39:56.657869 24909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:39:56.659044 24909 hybrid_clock.cc:648] HybridClock initialized: now 1786502396659008 us; error 48 us; skew 500 ppm
Aug 12 02:39:56 dist-test-slave-rmhg krb5kdc[22919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:39:56.662956 24909 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:39:56.664595 24909 webserver.cc:460] Webserver started at http://127.3.127.130:42805/ using document root <none> and password file <none>
I20260812 02:39:56.665514 24909 fs_manager.cc:359] Metadata directory not provided
I20260812 02:39:56.665709 24909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 02:39:56 dist-test-slave-rmhg krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
I20260812 02:39:56.693037 24909 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:39:56.696648 24931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:56.698659 24909 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260812 02:39:56.698822 24909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "63099c0e60854c838ea93538c1e33ad0"
format_stamp: "Formatted at 2026-08-12 02:39:55 on dist-test-slave-rmhg"
server_key: "a15953273f1f200ac4bee58c955e0cd2"
server_key_iv: "6fef3809eef3384d0e5639a068e7b0b6"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:56.699623 24909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:39:56.746074 24909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:39:56.750245 24909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:39:56.750473 24909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:39:56.751291 24909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:39:56.752482 24909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:39:56.752562 24909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:56.752624 24909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:39:56.752656 24909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:56.764477 24909 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:35575
I20260812 02:39:56.764878 25123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:35575 every 8 connection(s)
I20260812 02:39:56.765931 24909 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:39:56.775812 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 24909
I20260812 02:39:56.776402 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.127.148:52931/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:41887
--local_ip_for_outbound_sockets=127.3.127.131
--tserver_master_addrs=127.3.127.190:45045
--webserver_port=42205
--webserver_interface=127.3.127.131
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.127.148:52931/kms with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:39:56 dist-test-slave-rmhg krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:39:56.783675 25131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:56.768810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:45045 (local address 127.3.127.130:42185)
0812 02:39:56.769382 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:56.769440 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 02:39:56.770530 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:39:56.771136 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:39:56.771145 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:39:56.771500 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:39:56.772177 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:56.772195 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:56.773017 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:56.773020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:56.774427 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:56.774440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:56.775003 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:56.775008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:56.775222 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:56.775968 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:39:56.775991 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:39:56.777626 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:39:56.779722 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:39:56.779731 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:39:56.779752 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:39:56.780168 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:39:56.780542 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:39:56.780545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:39:56.780547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:39:56.780762 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:39:56.781408 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:39:56.781415 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:39:56.781813 (+ 398us) client_negotiation.cc:769] Sending connection context
0812 02:39:56.782045 (+ 232us) client_negotiation.cc:240] Negotiation successful
0812 02:39:56.782457 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":146,"threads_started":1}
I20260812 02:39:56.784499 24907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:56.768284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:42185 (local address 127.3.127.190:45045)
0812 02:39:56.768445 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:56.768450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:56.769561 (+ 1111us) server_negotiation.cc:407] Connection header received
0812 02:39:56.770786 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:56.770789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:56.770846 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:56.770925 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:39:56.772360 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:56.772897 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:56.774576 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:56.774823 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:56.774923 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:56.777871 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:39:56.777876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:39:56.777878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:39:56.777912 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:39:56.779569 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:56.780307 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:56.780311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:56.780313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:56.780425 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:56.780880 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:56.780883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:56.780885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:56.781204 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:39:56.781342 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:56.784144 (+ 2802us) server_negotiation.cc:299] Negotiation successful
0812 02:39:56.784318 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 02:39:56.786427 25125 heartbeater.cc:347] Connected to a master server at 127.3.127.190:45045
I20260812 02:39:56.786738 25125 heartbeater.cc:464] Registering TS with master...
I20260812 02:39:56.787415 25125 heartbeater.cc:511] Master 127.3.127.190:45045 requested a full tablet report, sending...
W20260812 02:39:56.789494 25125 heartbeater.cc:644] Failed to heartbeat to 127.3.127.190:45045 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 02:39:56.789633 25125 heartbeater.cc:464] Registering TS with master...
I20260812 02:39:56.789831 25125 heartbeater.cc:511] Master 127.3.127.190:45045 requested a full tablet report, sending...
I20260812 02:39:56.790795 25125 heartbeater.cc:464] Registering TS with master...
I20260812 02:39:56.790985 25125 heartbeater.cc:511] Master 127.3.127.190:45045 requested a full tablet report, sending...
W20260812 02:39:56.791877 25125 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 02:39:56.932407 25132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:39:56.932893 25132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:39:56.933019 25132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:39:56.938239 25132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:39:56.938315 25132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:39:56.938401 25132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:39:56.946130 25132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.127.148:52931/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:41887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=42205
--tserver_master_addrs=127.3.127.190:45045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:39:56.947405 25132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:39:56.948908 25132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:39:56.962881 25143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:39:56.962970 25149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:39:56.963454 25132 server_base.cc:1034] running on GCE node
W20260812 02:39:56.963446 25142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:39:56.964182 25132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:39:56.964936 25132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:39:56.966121 25132 hybrid_clock.cc:648] HybridClock initialized: now 1786502396966092 us; error 74 us; skew 500 ppm
Aug 12 02:39:56 dist-test-slave-rmhg krb5kdc[22919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:39:56.969681 25132 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:39:56.970944 25132 webserver.cc:460] Webserver started at http://127.3.127.131:42205/ using document root <none> and password file <none>
I20260812 02:39:56.971779 25132 fs_manager.cc:359] Metadata directory not provided
I20260812 02:39:56.971917 25132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 02:39:56 dist-test-slave-rmhg krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
I20260812 02:39:56.989933 25132 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:39:56.993364 25159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:56.998363 25132 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260812 02:39:56.998550 25132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "c701c28f53d7412f9c787b7d556e0874"
format_stamp: "Formatted at 2026-08-12 02:39:55 on dist-test-slave-rmhg"
server_key: "7120b330a2a91a1310a9355b383d6a7f"
server_key_iv: "3b8c54bffd2a839a79f16358e3702d21"
server_key_version: "kuduclusterkey@0"
I20260812 02:39:56.999189 25132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:39:57.021934 25132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:39:57.025504 25132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:39:57.025707 25132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:39:57.026355 25132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:39:57.027261 25132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:39:57.027309 25132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:57.027354 25132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:39:57.027381 25132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:39:57.036485 25132 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:41887
I20260812 02:39:57.036716 25365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:41887 every 8 connection(s)
I20260812 02:39:57.037706 25132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:39:57.047855 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 25132
Aug 12 02:39:57 dist-test-slave-rmhg krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:39:57.056627 24907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:57.039979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:41841 (local address 127.3.127.190:45045)
0812 02:39:57.040139 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:57.040143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:57.041072 (+ 929us) server_negotiation.cc:407] Connection header received
0812 02:39:57.042041 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:57.042044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:57.042130 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:57.042238 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:39:57.043618 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.044499 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.046650 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.047022 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.047111 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:57.051542 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:39:57.051547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:39:57.051550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:39:57.051584 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:39:57.053953 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:57.054721 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:57.054725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:57.054727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:57.054822 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:57.055268 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:57.055272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:57.055274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:57.055501 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:39:57.055585 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:57.056319 (+ 734us) server_negotiation.cc:299] Negotiation successful
0812 02:39:57.056458 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 02:39:57.057751 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:57.040392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:45045 (local address 127.3.127.131:41841)
0812 02:39:57.040885 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:57.040939 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 02:39:57.041831 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:39:57.042447 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:39:57.042457 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:39:57.042746 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:39:57.043387 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:57.043409 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.044656 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.044660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:57.046472 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:57.046500 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.047346 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.047350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:57.047586 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:57.048297 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:39:57.048320 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:39:57.051314 (+ 2994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:39:57.054112 (+ 2798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:39:57.054121 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:39:57.054143 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:39:57.054576 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:39:57.054938 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:39:57.054941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:39:57.054943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:39:57.055153 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:39:57.055674 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:39:57.055683 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:39:57.056042 (+ 359us) client_negotiation.cc:769] Sending connection context
0812 02:39:57.056243 (+ 201us) client_negotiation.cc:240] Negotiation successful
0812 02:39:57.056611 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":119,"threads_started":1}
I20260812 02:39:57.059257 25366 heartbeater.cc:347] Connected to a master server at 127.3.127.190:45045
I20260812 02:39:57.059512 25366 heartbeater.cc:464] Registering TS with master...
I20260812 02:39:57.060045 25366 heartbeater.cc:511] Master 127.3.127.190:45045 requested a full tablet report, sending...
I20260812 02:39:57.061699 24578 ts_manager.cc:194] Registered new tserver with Master: c701c28f53d7412f9c787b7d556e0874 (127.3.127.131:41887)
I20260812 02:39:57.063375 24578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:41841
I20260812 02:39:57.067675 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:57.049406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58070 (local address 127.3.127.190:45045)
0812 02:39:57.050771 (+ 1365us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:57.050780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:57.050793 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:39:57.050839 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:57.050842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:57.050890 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:57.051544 (+ 654us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:39:57.052629 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.053590 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.054842 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.055076 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.055167 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:57.057267 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:39:57.057271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:39:57.057274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:39:57.057300 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:39:57.059094 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:57.060613 (+ 1519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:57.060617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:57.060619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:57.060719 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:57.065090 (+ 4371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:57.065093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:57.065095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:57.065384 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:39:57.065587 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:57.067335 (+ 1748us) server_negotiation.cc:299] Negotiation successful
0812 02:39:57.067490 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1285,"thread_start_us":136,"threads_started":1}
Aug 12 02:39:57 dist-test-slave-rmhg krb5kdc[22919](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786502397, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:39:57.507203 24902 heartbeater.cc:464] Registering TS with master...
I20260812 02:39:57.507586 24902 heartbeater.cc:511] Master 127.3.127.190:45045 requested a full tablet report, sending...
I20260812 02:39:57.508715 24578 ts_manager.cc:194] Registered new tserver with Master: e12deda4f6e74e8ca67fced1490efd69 (127.3.127.129:33727)
I20260812 02:39:57.509615 24578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:58705
Aug 12 02:39:57 dist-test-slave-rmhg krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502397, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for HTTP/127.3.127.148@KRBTEST.COM
I20260812 02:39:57.792131 25125 heartbeater.cc:464] Registering TS with master...
I20260812 02:39:57.792481 25125 heartbeater.cc:511] Master 127.3.127.190:45045 requested a full tablet report, sending...
I20260812 02:39:57.793623 24578 ts_manager.cc:194] Registered new tserver with Master: 63099c0e60854c838ea93538c1e33ad0 (127.3.127.130:35575)
I20260812 02:39:57.794425 24578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:42185
I20260812 02:39:57.799598 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:39:57.818964 25397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:57.805726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.3.127.190:45045)
0812 02:39:57.806684 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:57.806689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:57.806984 (+ 295us) server_negotiation.cc:407] Connection header received
0812 02:39:57.808987 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:57.808991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:57.809067 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:57.809143 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:39:57.810418 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.811224 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.812421 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.812808 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.812939 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:57.814775 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:39:57.814780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:39:57.814783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:39:57.814820 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:39:57.816666 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:57.817303 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:57.817308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:57.817310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:57.817421 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:39:57.817780 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:39:57.817784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:39:57.817786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:39:57.818049 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:39:57.818181 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:57.818606 (+ 425us) server_negotiation.cc:299] Negotiation successful
0812 02:39:57.818767 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":135,"threads_started":1}
I20260812 02:39:57.825067 24578 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:39:57.830441 24578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:39:57.852926 25408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:57.845339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:35575 (local address 127.0.0.1:37854)
0812 02:39:57.847018 (+ 1679us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:57.847030 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:39:57.847116 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:39:57.847676 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:39:57.847684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:39:57.847710 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:39:57.847875 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:57.847883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.849566 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.849570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:57.851046 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:57.851058 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.852311 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.852315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:57.852460 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:57.852516 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 02:39:57.852617 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 02:39:57.852761 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1618,"thread_start_us":103,"threads_started":1}
I20260812 02:39:57.853946 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:57.845484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37854 (local address 127.3.127.130:35575)
0812 02:39:57.845791 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:57.845799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:57.847141 (+ 1342us) server_negotiation.cc:407] Connection header received
0812 02:39:57.847240 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:57.847246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:57.847484 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:57.848044 (+ 560us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:39:57.848196 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.849433 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.851233 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.852162 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.853437 (+ 1275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:57.853627 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:57.853682 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 02:39:57.853792 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
I20260812 02:39:57.854754 25411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:57.844220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:33727 (local address 127.0.0.1:47544)
0812 02:39:57.846574 (+ 2354us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:57.846636 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 02:39:57.846786 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:39:57.847850 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:39:57.847853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:39:57.847875 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:39:57.847979 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:57.847986 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.850617 (+ 2631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.850621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:57.852831 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:57.852843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.854359 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.854362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:57.854458 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:57.854484 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 02:39:57.854532 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 02:39:57.854620 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2135,"thread_start_us":134,"threads_started":1}
I20260812 02:39:57.855719 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:57.845245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47544 (local address 127.3.127.129:33727)
0812 02:39:57.847359 (+ 2114us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:57.847366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:57.847389 (+ 23us) server_negotiation.cc:407] Connection header received
0812 02:39:57.847465 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:57.847470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:57.847689 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:57.848164 (+ 475us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:39:57.849348 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.850478 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.852998 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.854227 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.855230 (+ 1003us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:57.855403 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:57.855461 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 02:39:57.855574 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1999,"thread_start_us":144,"threads_started":1}
I20260812 02:39:57.857630 25019 tablet_service.cc:1467] Processing CreateTablet for tablet bd547f0ac30d42498f388cd0d6ef66d9 (DEFAULT_TABLE table=test-table [id=14f139c3f6f848dba168125b28bb7fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:39:57.858680 25019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd547f0ac30d42498f388cd0d6ef66d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:39:57.860131 24787 tablet_service.cc:1467] Processing CreateTablet for tablet bd547f0ac30d42498f388cd0d6ef66d9 (DEFAULT_TABLE table=test-table [id=14f139c3f6f848dba168125b28bb7fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:39:57.862612 24787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd547f0ac30d42498f388cd0d6ef66d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:39:57.872486 25405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:57.845681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:41887 (local address 127.0.0.1:33670)
0812 02:39:57.850345 (+ 4664us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:57.850360 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:39:57.850463 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:39:57.853979 (+ 3516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:39:57.853982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:39:57.854007 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:39:57.854119 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:57.854125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.869241 (+ 15116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.869246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:57.870482 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:57.870500 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.872006 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.872011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:57.872116 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:57.872146 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 02:39:57.872222 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 02:39:57.872316 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4590,"thread_start_us":85,"threads_started":1}
I20260812 02:39:57.873963 25412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:57.845955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33670 (local address 127.3.127.131:41887)
0812 02:39:57.853398 (+ 7443us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:57.853407 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:57.853437 (+ 30us) server_negotiation.cc:407] Connection header received
0812 02:39:57.853531 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:57.853538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:57.853823 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:57.854019 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:39:57.861315 (+ 7296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.862613 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.870701 (+ 8088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.871626 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.871750 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:57.871929 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:57.873197 (+ 1268us) server_negotiation.cc:299] Negotiation successful
0812 02:39:57.873806 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7301,"thread_start_us":111,"threads_started":1}
I20260812 02:39:57.877365 25257 tablet_service.cc:1467] Processing CreateTablet for tablet bd547f0ac30d42498f388cd0d6ef66d9 (DEFAULT_TABLE table=test-table [id=14f139c3f6f848dba168125b28bb7fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:39:57.878691 25257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd547f0ac30d42498f388cd0d6ef66d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:39:57.881599 25414 tablet_bootstrap.cc:492] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69: Bootstrap starting.
I20260812 02:39:57.877915 25413 tablet_bootstrap.cc:492] T bd547f0ac30d42498f388cd0d6ef66d9 P 63099c0e60854c838ea93538c1e33ad0: Bootstrap starting.
I20260812 02:39:57.884533 25413 tablet_bootstrap.cc:654] T bd547f0ac30d42498f388cd0d6ef66d9 P 63099c0e60854c838ea93538c1e33ad0: Neither blocks nor log segments found. Creating new log.
I20260812 02:39:57.887331 25414 tablet_bootstrap.cc:654] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69: Neither blocks nor log segments found. Creating new log.
I20260812 02:39:57.888409 25414 log.cc:824] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69: Log is configured to *not* fsync() on all Append() calls
I20260812 02:39:57.889533 25413 log.cc:824] T bd547f0ac30d42498f388cd0d6ef66d9 P 63099c0e60854c838ea93538c1e33ad0: Log is configured to *not* fsync() on all Append() calls
I20260812 02:39:57.893008 25414 tablet_bootstrap.cc:492] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69: No bootstrap required, opened a new log
I20260812 02:39:57.893359 25414 ts_tablet_manager.cc:1397] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260812 02:39:57.897794 25414 raft_consensus.cc:348] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12deda4f6e74e8ca67fced1490efd69" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33727 } } peers { permanent_uuid: "63099c0e60854c838ea93538c1e33ad0" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35575 } } peers { permanent_uuid: "c701c28f53d7412f9c787b7d556e0874" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41887 } }
I20260812 02:39:57.898937 25421 tablet_bootstrap.cc:492] T bd547f0ac30d42498f388cd0d6ef66d9 P c701c28f53d7412f9c787b7d556e0874: Bootstrap starting.
I20260812 02:39:57.900135 25413 tablet_bootstrap.cc:492] T bd547f0ac30d42498f388cd0d6ef66d9 P 63099c0e60854c838ea93538c1e33ad0: No bootstrap required, opened a new log
I20260812 02:39:57.900506 25413 ts_tablet_manager.cc:1397] T bd547f0ac30d42498f388cd0d6ef66d9 P 63099c0e60854c838ea93538c1e33ad0: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20260812 02:39:57.906039 25413 raft_consensus.cc:348] T bd547f0ac30d42498f388cd0d6ef66d9 P 63099c0e60854c838ea93538c1e33ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12deda4f6e74e8ca67fced1490efd69" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33727 } } peers { permanent_uuid: "63099c0e60854c838ea93538c1e33ad0" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35575 } } peers { permanent_uuid: "c701c28f53d7412f9c787b7d556e0874" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41887 } }
I20260812 02:39:57.906572 25414 raft_consensus.cc:374] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:39:57.906713 25414 raft_consensus.cc:729] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e12deda4f6e74e8ca67fced1490efd69, State: Initialized, Role: FOLLOWER
I20260812 02:39:57.907510 25414 consensus_queue.cc:260] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12deda4f6e74e8ca67fced1490efd69" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33727 } } peers { permanent_uuid: "63099c0e60854c838ea93538c1e33ad0" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35575 } } peers { permanent_uuid: "c701c28f53d7412f9c787b7d556e0874" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41887 } }
I20260812 02:39:57.909124 25413 raft_consensus.cc:374] T bd547f0ac30d42498f388cd0d6ef66d9 P 63099c0e60854c838ea93538c1e33ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:39:57.909348 25413 raft_consensus.cc:729] T bd547f0ac30d42498f388cd0d6ef66d9 P 63099c0e60854c838ea93538c1e33ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63099c0e60854c838ea93538c1e33ad0, State: Initialized, Role: FOLLOWER
I20260812 02:39:57.910346 25414 ts_tablet_manager.cc:1428] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260812 02:39:57.910790 25413 consensus_queue.cc:260] T bd547f0ac30d42498f388cd0d6ef66d9 P 63099c0e60854c838ea93538c1e33ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12deda4f6e74e8ca67fced1490efd69" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33727 } } peers { permanent_uuid: "63099c0e60854c838ea93538c1e33ad0" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35575 } } peers { permanent_uuid: "c701c28f53d7412f9c787b7d556e0874" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41887 } }
I20260812 02:39:57.912233 25413 ts_tablet_manager.cc:1428] T bd547f0ac30d42498f388cd0d6ef66d9 P 63099c0e60854c838ea93538c1e33ad0: Time spent starting tablet: real 0.012s user 0.004s sys 0.003s
I20260812 02:39:57.912760 24902 heartbeater.cc:503] Master 127.3.127.190:45045 was elected leader, sending a full tablet report...
I20260812 02:39:57.917666 25125 heartbeater.cc:503] Master 127.3.127.190:45045 was elected leader, sending a full tablet report...
I20260812 02:39:57.918073 25421 tablet_bootstrap.cc:654] T bd547f0ac30d42498f388cd0d6ef66d9 P c701c28f53d7412f9c787b7d556e0874: Neither blocks nor log segments found. Creating new log.
I20260812 02:39:57.919344 25421 log.cc:824] T bd547f0ac30d42498f388cd0d6ef66d9 P c701c28f53d7412f9c787b7d556e0874: Log is configured to *not* fsync() on all Append() calls
I20260812 02:39:57.931890 25421 tablet_bootstrap.cc:492] T bd547f0ac30d42498f388cd0d6ef66d9 P c701c28f53d7412f9c787b7d556e0874: No bootstrap required, opened a new log
I20260812 02:39:57.932186 25421 ts_tablet_manager.cc:1397] T bd547f0ac30d42498f388cd0d6ef66d9 P c701c28f53d7412f9c787b7d556e0874: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.000s
I20260812 02:39:57.939357 25421 raft_consensus.cc:348] T bd547f0ac30d42498f388cd0d6ef66d9 P c701c28f53d7412f9c787b7d556e0874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12deda4f6e74e8ca67fced1490efd69" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33727 } } peers { permanent_uuid: "63099c0e60854c838ea93538c1e33ad0" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35575 } } peers { permanent_uuid: "c701c28f53d7412f9c787b7d556e0874" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41887 } }
I20260812 02:39:57.939718 25421 raft_consensus.cc:374] T bd547f0ac30d42498f388cd0d6ef66d9 P c701c28f53d7412f9c787b7d556e0874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:39:57.939782 25421 raft_consensus.cc:729] T bd547f0ac30d42498f388cd0d6ef66d9 P c701c28f53d7412f9c787b7d556e0874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c701c28f53d7412f9c787b7d556e0874, State: Initialized, Role: FOLLOWER
I20260812 02:39:57.940315 25421 consensus_queue.cc:260] T bd547f0ac30d42498f388cd0d6ef66d9 P c701c28f53d7412f9c787b7d556e0874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12deda4f6e74e8ca67fced1490efd69" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33727 } } peers { permanent_uuid: "63099c0e60854c838ea93538c1e33ad0" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35575 } } peers { permanent_uuid: "c701c28f53d7412f9c787b7d556e0874" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41887 } }
I20260812 02:39:57.941051 25423 raft_consensus.cc:482] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:39:57.941290 25423 raft_consensus.cc:504] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12deda4f6e74e8ca67fced1490efd69" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33727 } } peers { permanent_uuid: "63099c0e60854c838ea93538c1e33ad0" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35575 } } peers { permanent_uuid: "c701c28f53d7412f9c787b7d556e0874" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41887 } }
I20260812 02:39:57.941799 25421 ts_tablet_manager.cc:1428] T bd547f0ac30d42498f388cd0d6ef66d9 P c701c28f53d7412f9c787b7d556e0874: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260812 02:39:57.942313 25366 heartbeater.cc:503] Master 127.3.127.190:45045 was elected leader, sending a full tablet report...
I20260812 02:39:57.942658 25423 leader_election.cc:288] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63099c0e60854c838ea93538c1e33ad0 (127.3.127.130:35575), c701c28f53d7412f9c787b7d556e0874 (127.3.127.131:41887)
I20260812 02:39:57.949048 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:57.943364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:42243 (local address 127.3.127.130:35575)
0812 02:39:57.943522 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:57.943527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:57.943971 (+ 444us) server_negotiation.cc:407] Connection header received
0812 02:39:57.944223 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:57.944227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:57.944281 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:57.944329 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:39:57.944856 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.945685 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.947467 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.948064 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.948221 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:57.948286 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:57.948769 (+ 483us) server_negotiation.cc:299] Negotiation successful
0812 02:39:57.948878 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 02:39:57.949615 25433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:57.943183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:35575 (local address 127.3.127.129:42243)
0812 02:39:57.943862 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:57.943880 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:39:57.944089 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:39:57.944494 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:39:57.944497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:39:57.944518 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:39:57.944717 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:57.944724 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.945839 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.945842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:57.947311 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:57.947325 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.948413 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.948417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:57.948589 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:57.948622 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 02:39:57.949337 (+ 715us) client_negotiation.cc:240] Negotiation successful
0812 02:39:57.949429 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":223,"threads_started":1}
I20260812 02:39:57.950379 25050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd547f0ac30d42498f388cd0d6ef66d9" candidate_uuid: "e12deda4f6e74e8ca67fced1490efd69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63099c0e60854c838ea93538c1e33ad0" is_pre_election: true
I20260812 02:39:57.950793 25050 raft_consensus.cc:2454] T bd547f0ac30d42498f388cd0d6ef66d9 P 63099c0e60854c838ea93538c1e33ad0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e12deda4f6e74e8ca67fced1490efd69 in term 0.
I20260812 02:39:57.951344 24717 leader_election.cc:302] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63099c0e60854c838ea93538c1e33ad0, e12deda4f6e74e8ca67fced1490efd69; no voters:
I20260812 02:39:57.951720 25423 raft_consensus.cc:2781] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:39:57.951815 25423 raft_consensus.cc:482] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:39:57.951858 25423 raft_consensus.cc:3037] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:39:57.953624 25423 raft_consensus.cc:504] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12deda4f6e74e8ca67fced1490efd69" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33727 } } peers { permanent_uuid: "63099c0e60854c838ea93538c1e33ad0" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35575 } } peers { permanent_uuid: "c701c28f53d7412f9c787b7d556e0874" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41887 } }
I20260812 02:39:57.954039 25423 leader_election.cc:288] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69 [CANDIDATE]: Term 1 election: Requested vote from peers 63099c0e60854c838ea93538c1e33ad0 (127.3.127.130:35575), c701c28f53d7412f9c787b7d556e0874 (127.3.127.131:41887)
I20260812 02:39:57.954535 25050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd547f0ac30d42498f388cd0d6ef66d9" candidate_uuid: "e12deda4f6e74e8ca67fced1490efd69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63099c0e60854c838ea93538c1e33ad0"
I20260812 02:39:57.954687 25050 raft_consensus.cc:3037] T bd547f0ac30d42498f388cd0d6ef66d9 P 63099c0e60854c838ea93538c1e33ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:39:57.956198 25050 raft_consensus.cc:2454] T bd547f0ac30d42498f388cd0d6ef66d9 P 63099c0e60854c838ea93538c1e33ad0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e12deda4f6e74e8ca67fced1490efd69 in term 1.
I20260812 02:39:57.956599 24717 leader_election.cc:302] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63099c0e60854c838ea93538c1e33ad0, e12deda4f6e74e8ca67fced1490efd69; no voters:
I20260812 02:39:57.957249 25423 raft_consensus.cc:2781] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:39:57.957561 25423 raft_consensus.cc:686] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69 [term 1 LEADER]: Becoming Leader. State: Replica: e12deda4f6e74e8ca67fced1490efd69, State: Running, Role: LEADER
I20260812 02:39:57.957995 25423 consensus_queue.cc:237] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12deda4f6e74e8ca67fced1490efd69" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33727 } } peers { permanent_uuid: "63099c0e60854c838ea93538c1e33ad0" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35575 } } peers { permanent_uuid: "c701c28f53d7412f9c787b7d556e0874" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41887 } }
I20260812 02:39:57.961222 25412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:57.944838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:40769 (local address 127.3.127.131:41887)
0812 02:39:57.944961 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:57.944965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:57.950468 (+ 5503us) server_negotiation.cc:407] Connection header received
0812 02:39:57.950575 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:57.950578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:57.950635 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:57.950724 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:39:57.957323 (+ 6599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.958165 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.959738 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.960367 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.960514 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:57.960573 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:57.960900 (+ 327us) server_negotiation.cc:299] Negotiation successful
0812 02:39:57.960999 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 02:39:57.961674 25433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:57.943713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:41887 (local address 127.3.127.129:40769)
0812 02:39:57.950355 (+ 6642us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:57.950371 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:39:57.950477 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:39:57.952721 (+ 2244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:39:57.952725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:39:57.952752 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:39:57.952859 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:57.952865 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.958347 (+ 5482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.958351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:57.959570 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:39:57.959586 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.960646 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.960650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:39:57.960742 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:57.960768 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 02:39:57.961447 (+ 679us) client_negotiation.cc:240] Negotiation successful
0812 02:39:57.961545 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6580,"thread_start_us":140,"threads_started":1}
I20260812 02:39:57.961989 24578 catalog_manager.cc:5495] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69 reported cstate change: term changed from 0 to 1, leader changed from <none> to e12deda4f6e74e8ca67fced1490efd69 (127.3.127.129). New cstate: current_term: 1 leader_uuid: "e12deda4f6e74e8ca67fced1490efd69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12deda4f6e74e8ca67fced1490efd69" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 33727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63099c0e60854c838ea93538c1e33ad0" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c701c28f53d7412f9c787b7d556e0874" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41887 } health_report { overall_health: UNKNOWN } } }
I20260812 02:39:57.962569 25283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd547f0ac30d42498f388cd0d6ef66d9" candidate_uuid: "e12deda4f6e74e8ca67fced1490efd69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c701c28f53d7412f9c787b7d556e0874" is_pre_election: true
I20260812 02:39:57.962903 25283 raft_consensus.cc:2454] T bd547f0ac30d42498f388cd0d6ef66d9 P c701c28f53d7412f9c787b7d556e0874 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e12deda4f6e74e8ca67fced1490efd69 in term 0.
I20260812 02:39:57.963285 25281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd547f0ac30d42498f388cd0d6ef66d9" candidate_uuid: "e12deda4f6e74e8ca67fced1490efd69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c701c28f53d7412f9c787b7d556e0874"
I20260812 02:39:57.963408 25281 raft_consensus.cc:3037] T bd547f0ac30d42498f388cd0d6ef66d9 P c701c28f53d7412f9c787b7d556e0874 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:39:57.965030 25281 raft_consensus.cc:2454] T bd547f0ac30d42498f388cd0d6ef66d9 P c701c28f53d7412f9c787b7d556e0874 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e12deda4f6e74e8ca67fced1490efd69 in term 1.
W20260812 02:39:57.979043 24905 tablet.cc:2369] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:39:57.994439 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:57.988765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47556 (local address 127.3.127.129:33727)
0812 02:39:57.988901 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 02:39:57.988906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:39:57.988921 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:39:57.989025 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:39:57.989028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:39:57.989105 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:39:57.989273 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:39:57.989715 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.990539 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.991827 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:39:57.992180 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:39:57.992293 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:39:57.992795 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:39:57.993534 (+ 739us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:39:57.993675 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 02:39:57.994207 (+ 532us) server_negotiation.cc:299] Negotiation successful
0812 02:39:57.994282 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 02:39:58.003429 25281 raft_consensus.cc:1261] T bd547f0ac30d42498f388cd0d6ef66d9 P c701c28f53d7412f9c787b7d556e0874 [term 1 FOLLOWER]: Refusing update from remote peer e12deda4f6e74e8ca67fced1490efd69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:39:58.004379 25423 consensus_queue.cc:1035] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c701c28f53d7412f9c787b7d556e0874" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:39:58.005357 25050 raft_consensus.cc:1261] T bd547f0ac30d42498f388cd0d6ef66d9 P 63099c0e60854c838ea93538c1e33ad0 [term 1 FOLLOWER]: Refusing update from remote peer e12deda4f6e74e8ca67fced1490efd69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:39:58.006114 25423 consensus_queue.cc:1035] T bd547f0ac30d42498f388cd0d6ef66d9 P e12deda4f6e74e8ca67fced1490efd69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63099c0e60854c838ea93538c1e33ad0" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 35575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:39:58.016518 25439 mvcc.cc:205] Tried to move back new op lower bound from 7317513822211031040 to 7317513822039425024. Current Snapshot: MvccSnapshot[applied={T|T < 7317513822211031040}]
I20260812 02:39:58.016346 25441 mvcc.cc:205] Tried to move back new op lower bound from 7317513822211031040 to 7317513822039425024. Current Snapshot: MvccSnapshot[applied={T|T < 7317513822211031040}]
I20260812 02:39:58.024863 25442 mvcc.cc:205] Tried to move back new op lower bound from 7317513822211031040 to 7317513822039425024. Current Snapshot: MvccSnapshot[applied={T|T < 7317513822211031040}]
I20260812 02:39:58.033198 24574 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:39:58.033413 24574 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:39:58.036891 24574 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5748ff60a4dc4112a4de838288d6dc81: Sending DeleteTablet for 3 replicas of tablet bd547f0ac30d42498f388cd0d6ef66d9
I20260812 02:39:58.038371 25019 tablet_service.cc:1514] Processing DeleteTablet for tablet bd547f0ac30d42498f388cd0d6ef66d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:39:58 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:37854
I20260812 02:39:58.039153 25257 tablet_service.cc:1514] Processing DeleteTablet for tablet bd547f0ac30d42498f388cd0d6ef66d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:39:58 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:33670
I20260812 02:39:58.040261 24787 tablet_service.cc:1514] Processing DeleteTablet for tablet bd547f0ac30d42498f388cd0d6ef66d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:39:58 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:47544
I20260812 02:39:58.041685 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 24642
I20260812 02:39:58.064766 25454 tablet_replica.cc:330] T bd547f0ac30d42498f388cd0d6ef66d9 P 63099c0e60854c838ea93538c1e33ad0: stopping tablet replica
I20260812 02:39:58.065384 25454 raft_consensus.cc:2229] T bd547f0ac30d42498f388cd0d6ef66d9 P 63099c0e60854c838ea93538c1e33ad0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:39:58.065977 25454 raft_consensus.cc:2258] T bd547f0ac30d42498f388cd0d6ef66d9 P 63099c0e60854c838ea93538c1e33ad0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:39:58.068055 25454 ts_tablet_manager.cc:1905] T bd547f0ac30d42498f388cd0d6ef66d9 P 63099c0e60854c838ea93538c1e33ad0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:39:58.071754 25454 ts_tablet_manager.cc:1918] T bd547f0ac30d42498f388cd0d6ef66d9 P 63099c0e60854c838ea93538c1e33ad0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:39:58.071983 25454 log.cc:1196] T bd547f0ac30d42498f388cd0d6ef66d9 P 63099c0e60854c838ea93538c1e33ad0: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/bd547f0ac30d42498f388cd0d6ef66d9
I20260812 02:39:58.072413 25454 ts_tablet_manager.cc:1939] T bd547f0ac30d42498f388cd0d6ef66d9 P 63099c0e60854c838ea93538c1e33ad0: Deleting consensus metadata
I20260812 02:39:58.074070 24550 catalog_manager.cc:4841] TS 63099c0e60854c838ea93538c1e33ad0 (127.3.127.130:35575): tablet bd547f0ac30d42498f388cd0d6ef66d9 (table test-table [id=14f139c3f6f848dba168125b28bb7fcd]) successfully deleted
I20260812 02:39:58.078573 25456 tablet_replica.cc:330] T bd547f0ac30d42498f388cd0d6ef66d9 P c701c28f53d7412f9c787b7d556e0874: stopping tablet replica
I20260812 02:39:58.078997 25456 raft_consensus.cc:2229] T bd547f0ac30d42498f388cd0d6ef66d9 P c701c28f53d7412f9c787b7d556e0874 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:39:58.079417 25456 raft_consensus.cc:2258] T bd547f0ac30d42498f388cd0d6ef66d9 P c701c28f53d7412f9c787b7d556e0874 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:39:58.081367 25456 ts_tablet_manager.cc:1905] T bd547f0ac30d42498f388cd0d6ef66d9 P c701c28f53d7412f9c787b7d556e0874: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 02:39:58.084347 24558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.127.129:33727 (error 108)
I20260812 02:39:58.085465 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 24909
I20260812 02:39:58.087754 25456 ts_tablet_manager.cc:1918] T bd547f0ac30d42498f388cd0d6ef66d9 P c701c28f53d7412f9c787b7d556e0874: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:39:58.093392 25456 log.cc:1196] T bd547f0ac30d42498f388cd0d6ef66d9 P c701c28f53d7412f9c787b7d556e0874: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/bd547f0ac30d42498f388cd0d6ef66d9
I20260812 02:39:58.093976 25456 ts_tablet_manager.cc:1939] T bd547f0ac30d42498f388cd0d6ef66d9 P c701c28f53d7412f9c787b7d556e0874: Deleting consensus metadata
I20260812 02:39:58.095708 24550 catalog_manager.cc:4841] TS c701c28f53d7412f9c787b7d556e0874 (127.3.127.131:41887): tablet bd547f0ac30d42498f388cd0d6ef66d9 (table test-table [id=14f139c3f6f848dba168125b28bb7fcd]) successfully deleted
I20260812 02:39:58.183158 25405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:39:58.182687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:33727 (local address 127.0.0.1:47568)
0812 02:39:58.182945 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0812 02:39:58.183046 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.129:33727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
W20260812 02:39:58.183442 24558 catalog_manager.cc:4584] TS e12deda4f6e74e8ca67fced1490efd69 (127.3.127.129:33727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd547f0ac30d42498f388cd0d6ef66d9: Network error: Client connection negotiation failed: client connection to 127.3.127.129:33727: connect: Connection refused (error 111)
I20260812 02:39:58.192988 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 25132
I20260812 02:39:58.216790 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 24529
I20260812 02:39:58.240788 3582 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 02:39:58.587829 3582 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 02:39:58.587980 3582 mini_ranger.cc:66] Stopping Ranger...
I20260812 02:39:58.943871 3582 mini_ranger.cc:68] Stopped Ranger
2026-08-12 02:39:58.944 UTC [22969] LOG: received smart shutdown request
2026-08-12 02:39:58.948 UTC [22969] LOG: background worker "logical replication launcher" (PID 22980) exited with exit code 1
2026-08-12 02:39:58.949 UTC [22974] LOG: shutting down
2026-08-12 02:39:59.111 UTC [22969] LOG: database system is shut down
2026-08-12T02:39:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52501 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:39:59 dist-test-slave-rmhg krb5kdc[25476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:39:59 dist-test-slave-rmhg krb5kdc[25476](info): set up 2 sockets
Aug 12 02:39:59 dist-test-slave-rmhg krb5kdc[25476](info): commencing operation
krb5kdc: starting...
W20260812 02:40:00.395545 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:40:00 dist-test-slave-rmhg krb5kdc[25476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:40:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:40:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:40:00.531611 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:38657
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:38657,127.3.127.189:44773,127.3.127.188:43277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:40:00.654345 25495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:00.654666 25495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:00.654726 25495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:00.658978 25495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:40:00.659062 25495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:00.659090 25495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:00.659113 25495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:40:00.659144 25495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:40:00.664930 25495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:38657,127.3.127.189:44773,127.3.127.188:43277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:38657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:00.666186 25495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:00.667299 25495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:00.676566 25505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:00.676553 25509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:00.676530 25506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:00.677349 25495 server_base.cc:1034] running on GCE node
I20260812 02:40:00.677949 25495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:00.679189 25495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:00.680384 25495 hybrid_clock.cc:648] HybridClock initialized: now 1786502400680365 us; error 39 us; skew 500 ppm
Aug 12 02:40:00 dist-test-slave-rmhg krb5kdc[25476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502400, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:00.683462 25495 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:40:00.684576 25495 webserver.cc:460] Webserver started at http://127.3.127.190:45533/ using document root <none> and password file <none>
I20260812 02:40:00.685220 25495 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:00.685299 25495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:00.685544 25495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:40:00.687846 25495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "1bf2561d19a14f498d4eaaccf8aed7f8"
format_stamp: "Formatted at 2026-08-12 02:40:00 on dist-test-slave-rmhg"
server_key: "c6173bbbe6bddaed219afb672461c5ae"
server_key_iv: "4423850ad48f77462b04442a177ed68a"
server_key_version: "encryptionkey@0"
I20260812 02:40:00.688508 25495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "1bf2561d19a14f498d4eaaccf8aed7f8"
format_stamp: "Formatted at 2026-08-12 02:40:00 on dist-test-slave-rmhg"
server_key: "c6173bbbe6bddaed219afb672461c5ae"
server_key_iv: "4423850ad48f77462b04442a177ed68a"
server_key_version: "encryptionkey@0"
I20260812 02:40:00.692776 25495 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:40:00.695711 25518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:00.697088 25495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:40:00.697283 25495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "1bf2561d19a14f498d4eaaccf8aed7f8"
format_stamp: "Formatted at 2026-08-12 02:40:00 on dist-test-slave-rmhg"
server_key: "c6173bbbe6bddaed219afb672461c5ae"
server_key_iv: "4423850ad48f77462b04442a177ed68a"
server_key_version: "encryptionkey@0"
I20260812 02:40:00.697407 25495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:00.723397 25495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:00.726938 25495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:00.727162 25495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:00.735033 25495 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:38657
I20260812 02:40:00.735098 25611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:38657 every 8 connection(s)
I20260812 02:40:00.736095 25495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:40:00.739225 25612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:40:00.740618 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 25495
I20260812 02:40:00.740774 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:40:00.741061 3582 external_mini_cluster.cc:1428] Setting key ec3d1191cc97f0c70bb0d14d0e4bef84
I20260812 02:40:00.743494 25612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.127.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.189@KRBTEST.COM" created.
Aug 12 02:40:00 dist-test-slave-rmhg krb5kdc[25476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502400, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:40:00.759910 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:00.747431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50958 (local address 127.3.127.190:38657)
0812 02:40:00.747760 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:00.747771 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:00.747819 (+ 48us) server_negotiation.cc:407] Connection header received
0812 02:40:00.748309 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:00.748314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:00.748518 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:00.748694 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:00.749626 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:00.750572 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:00.751721 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:00.752164 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:00.752368 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:00.755171 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:00.755177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:00.755189 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:00.755211 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:00.756698 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:00.757183 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:00.757189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:00.757191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:00.757309 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:00.757600 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:00.757603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:00.757605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:00.758024 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:00.758147 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:00.758542 (+ 395us) server_negotiation.cc:299] Negotiation successful
0812 02:40:00.758803 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":220,"thread_start_us":102,"threads_started":1}
I20260812 02:40:00.759907 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:00.746304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38657 (local address 127.0.0.1:50958)
0812 02:40:00.746953 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:00.747028 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:00.747993 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:00.748714 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:00.748720 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:00.749201 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:00.749475 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:00.749493 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:00.750695 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:00.750698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:00.751599 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:00.751611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:00.752317 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:00.752321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:00.752558 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:00.753264 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:00.753287 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:00.755010 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:00.756802 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:00.756810 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:00.756822 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:00.757054 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:00.757396 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:00.757400 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:00.757402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:00.757517 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:00.758133 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:00.758140 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:00.758289 (+ 149us) client_negotiation.cc:769] Sending connection context
0812 02:40:00.758452 (+ 163us) client_negotiation.cc:240] Negotiation successful
0812 02:40:00.758863 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":153,"thread_start_us":180,"threads_started":1}
I20260812 02:40:00.763280 25612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:40:00.764266 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:00.763872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:44773 (local address 127.0.0.1:49152)
0812 02:40:00.764064 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:00.764168 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.189:44773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260812 02:40:00.764570 25519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.127.189:44773: connect: Connection refused (error 111)
I20260812 02:40:00.765615 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:00.765275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:44773 (local address 127.0.0.1:49154)
0812 02:40:00.765401 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:00.765477 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.189:44773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260812 02:40:00.765784 25612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.189:44773: Network error: Client connection negotiation failed: client connection to 127.3.127.189:44773: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.189.keytab.
Entry for principal kudu/127.3.127.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.189.keytab.
I20260812 02:40:00.771878 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.189:44773
--webserver_interface=127.3.127.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.189.keytab
--principal=kudu/127.3.127.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:38657,127.3.127.189:44773,127.3.127.188:43277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:40:00.815057 25612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } attempt: 1
I20260812 02:40:00.815951 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:00.815581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:44773 (local address 127.0.0.1:49164)
0812 02:40:00.815773 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:00.815850 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.189:44773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260812 02:40:00.816735 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:00.816467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:44773 (local address 127.0.0.1:49180)
0812 02:40:00.816581 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:00.816642 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.189:44773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260812 02:40:00.816903 25612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.189:44773: Network error: Client connection negotiation failed: client connection to 127.3.127.189:44773: connect: Connection refused (error 111)
I20260812 02:40:00.885066 25612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } attempt: 2
I20260812 02:40:00.886051 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:00.885692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:44773 (local address 127.0.0.1:49192)
0812 02:40:00.885876 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:00.885952 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.189:44773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260812 02:40:00.886827 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:00.886548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:44773 (local address 127.0.0.1:49196)
0812 02:40:00.886642 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:00.886756 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.189:44773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260812 02:40:00.886965 25612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.189:44773: Network error: Client connection negotiation failed: client connection to 127.3.127.189:44773: connect: Connection refused (error 111)
W20260812 02:40:00.892421 25626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:00.892722 25626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:00.892766 25626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:00.896878 25626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:40:00.896960 25626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:00.896977 25626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:00.896992 25626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:40:00.897006 25626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:40:00.902752 25626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:38657,127.3.127.189:44773,127.3.127.188:43277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.189.keytab
--principal=kudu/127.3.127.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.189:44773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.127.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:00.903887 25626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:00.904908 25626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:00.913730 25638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:00.913724 25636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:00.913720 25634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:00.913796 25626 server_base.cc:1034] running on GCE node
I20260812 02:40:00.914610 25626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:00.915822 25626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:00.916985 25626 hybrid_clock.cc:648] HybridClock initialized: now 1786502400916970 us; error 39 us; skew 500 ppm
Aug 12 02:40:00 dist-test-slave-rmhg krb5kdc[25476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502400, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:00.919780 25626 init.cc:377] Logged in from keytab as kudu/127.3.127.189@KRBTEST.COM (short username kudu)
I20260812 02:40:00.920715 25626 webserver.cc:460] Webserver started at http://127.3.127.189:42199/ using document root <none> and password file <none>
I20260812 02:40:00.921312 25626 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:00.921387 25626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:00.921599 25626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:40:00.923679 25626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-1/data/instance:
uuid: "9aaac3eb29e249509b65e72aaaca820c"
format_stamp: "Formatted at 2026-08-12 02:40:00 on dist-test-slave-rmhg"
server_key: "176d519e4f80cb55d34751bc5cbdd6fe"
server_key_iv: "a3c1c8306cfaa69c547c6387a012ee29"
server_key_version: "encryptionkey@0"
I20260812 02:40:00.924259 25626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-1/wal/instance:
uuid: "9aaac3eb29e249509b65e72aaaca820c"
format_stamp: "Formatted at 2026-08-12 02:40:00 on dist-test-slave-rmhg"
server_key: "176d519e4f80cb55d34751bc5cbdd6fe"
server_key_iv: "a3c1c8306cfaa69c547c6387a012ee29"
server_key_version: "encryptionkey@0"
I20260812 02:40:00.928131 25626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 02:40:00.930867 25650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:00.932304 25626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:40:00.932456 25626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-1/wal
uuid: "9aaac3eb29e249509b65e72aaaca820c"
format_stamp: "Formatted at 2026-08-12 02:40:00 on dist-test-slave-rmhg"
server_key: "176d519e4f80cb55d34751bc5cbdd6fe"
server_key_iv: "a3c1c8306cfaa69c547c6387a012ee29"
server_key_version: "encryptionkey@0"
I20260812 02:40:00.932595 25626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:00.951089 25626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:00.954535 25626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:00.954758 25626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:00.962567 25626 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.189:44773
I20260812 02:40:00.962618 25738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.189:44773 every 8 connection(s)
I20260812 02:40:00.963673 25626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-1/data/info.pb
I20260812 02:40:00.966910 25741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:40:00.969864 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 25626
I20260812 02:40:00.970057 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-1/wal/instance
I20260812 02:40:00.970381 3582 external_mini_cluster.cc:1428] Setting key 3d477bb465aae17ff96d7b967697fcd4
I20260812 02:40:00.971908 25741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:40:00.978120 25612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } attempt: 3
WARNING: no policy specified for kudu/127.3.127.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.188@KRBTEST.COM" created.
Aug 12 02:40:00 dist-test-slave-rmhg krb5kdc[25476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502400, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for kudu/127.3.127.189@KRBTEST.COM
Aug 12 02:40:00 dist-test-slave-rmhg krb5kdc[25476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502400, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.189@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:40:00.989192 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:00.978609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:44773 (local address 127.0.0.1:49210)
0812 02:40:00.978782 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:00.978797 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:00.978879 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:00.979807 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:00.979810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:00.980010 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:00.980136 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:00.980143 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:00.981478 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:00.981481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:00.982097 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:00.982109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:00.982855 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:00.982859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:00.982969 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:00.983287 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:00.983300 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:00.984832 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:00.987189 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:00.987193 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:00.987195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:00.987450 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:00.987820 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:00.987823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:00.987825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:00.987914 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:00.988627 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:00.988630 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:00.988749 (+ 119us) client_negotiation.cc:769] Sending connection context
0812 02:40:00.988831 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 02:40:00.988981 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":25}
I20260812 02:40:00.989377 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:00.974293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.3.127.190:38657)
0812 02:40:00.974478 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:00.974483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:00.975551 (+ 1068us) server_negotiation.cc:407] Connection header received
0812 02:40:00.976966 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:00.976971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:00.977052 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:00.977136 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:00.979051 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:00.979639 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:00.981076 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:00.982174 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:00.982276 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:00.985744 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:00.985748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:00.985751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:00.985780 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:00.987278 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:00.987785 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:00.987788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:00.987789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:00.987877 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:00.988224 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:00.988227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:00.988229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:00.988447 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:00.988541 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:00.988984 (+ 443us) server_negotiation.cc:299] Negotiation successful
0812 02:40:00.989126 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":72}
I20260812 02:40:00.990896 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:00.978833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.3.127.189:44773)
0812 02:40:00.979255 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:00.979267 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:00.979293 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:40:00.979350 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:00.979356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:00.979594 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:00.979834 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:00.980271 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:00.981367 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:00.982210 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:00.982724 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:00.982851 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:00.984985 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:00.984991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:00.985004 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:00.985028 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:00.987051 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:00.987565 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:00.987572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:00.987573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:00.987701 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:00.988026 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:00.988029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:00.988031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:00.988505 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:00.989460 (+ 955us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:00.989579 (+ 119us) server_negotiation.cc:299] Negotiation successful
0812 02:40:00.989846 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":142,"server-negotiator.queue_time_us":308,"thread_start_us":178,"threads_started":1}
I20260812 02:40:00.990896 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:00.974673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38657 (local address 127.0.0.1:50962)
0812 02:40:00.975302 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:00.975388 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:00.976649 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:00.977436 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:00.977448 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:00.977975 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:00.978828 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:00.978855 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:00.979773 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:00.979776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:00.980948 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:00.980960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:00.982340 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:00.982345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:00.982633 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:00.983247 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:00.983271 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:00.985579 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:00.987370 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:00.987380 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:00.987392 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:00.987682 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:00.987980 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:00.987983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:00.987985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:00.988130 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:00.988555 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:00.988561 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:00.988718 (+ 157us) client_negotiation.cc:769] Sending connection context
0812 02:40:00.988903 (+ 185us) client_negotiation.cc:240] Negotiation successful
0812 02:40:00.989781 (+ 878us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":61,"thread_start_us":141,"threads_started":1}
I20260812 02:40:00.992724 25741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:40:00.993458 25612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:40:00.994164 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:00.993893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43277 (local address 127.0.0.1:35470)
0812 02:40:00.994026 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:00.994086 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 02:40:00.994948 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:00.994662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43277 (local address 127.0.0.1:35472)
0812 02:40:00.994826 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:00.994879 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260812 02:40:00.995177 25612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.188:43277: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43277: connect: Connection refused (error 111)
Aug 12 02:40:00 dist-test-slave-rmhg krb5kdc[25476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502400, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.189@KRBTEST.COM for kudu/127.3.127.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.188.keytab.
Entry for principal kudu/127.3.127.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.188.keytab.
I20260812 02:40:01.000890 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.188:43277
--webserver_interface=127.3.127.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.188.keytab
--principal=kudu/127.3.127.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:38657,127.3.127.189:44773,127.3.127.188:43277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:40:01.001554 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:00.993309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:44773 (local address 127.0.0.1:49222)
0812 02:40:00.993406 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:00.993421 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:00.993498 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:00.993677 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:00.993680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:00.993840 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:00.993928 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:00.993934 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:00.994821 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:00.994825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:00.995618 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:00.995629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:00.996212 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:00.996215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:00.996300 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:00.996769 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:00.996781 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:00.998247 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:00.999935 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:00.999938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:00.999940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:01.000126 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:01.000397 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:01.000400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:01.000402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:01.000498 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:01.000961 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:01.000964 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:01.001081 (+ 117us) client_negotiation.cc:769] Sending connection context
0812 02:40:01.001245 (+ 164us) client_negotiation.cc:240] Negotiation successful
0812 02:40:01.001388 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260812 02:40:01.002130 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:00.993375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49222 (local address 127.3.127.189:44773)
0812 02:40:00.993457 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:00.993460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:00.993482 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:40:00.993564 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:00.993567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:00.993605 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:00.993667 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:00.994051 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:00.994692 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:00.995751 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:00.996070 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:00.996164 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:00.998403 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:00.998407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:00.998409 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:00.998433 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:00.999826 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.000226 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.000229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.000230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.000311 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.000633 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.000637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.000638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.000860 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:01.000963 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:01.001873 (+ 910us) server_negotiation.cc:299] Negotiation successful
0812 02:40:01.001989 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 02:40:01.003032 25741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:40:01.003849 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.003562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43277 (local address 127.0.0.1:35484)
0812 02:40:01.003697 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:01.003786 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260812 02:40:01.004631 25657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43277: connect: Connection refused (error 111)
I20260812 02:40:01.005679 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.005437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43277 (local address 127.0.0.1:35500)
0812 02:40:01.005549 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:01.005612 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 02:40:01.005841 25741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.188:43277: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43277: connect: Connection refused (error 111)
I20260812 02:40:01.026366 25612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } attempt: 1
I20260812 02:40:01.027316 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.027011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43277 (local address 127.0.0.1:35512)
0812 02:40:01.027162 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:01.027230 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260812 02:40:01.028220 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.027961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43277 (local address 127.0.0.1:35522)
0812 02:40:01.028098 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:01.028152 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260812 02:40:01.028384 25612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.188:43277: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43277: connect: Connection refused (error 111)
I20260812 02:40:01.055135 25741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } attempt: 1
I20260812 02:40:01.056180 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.055792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43277 (local address 127.0.0.1:35532)
0812 02:40:01.055996 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:01.056086 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260812 02:40:01.056985 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.056758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43277 (local address 127.0.0.1:35536)
0812 02:40:01.056870 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:01.056920 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 02:40:01.057140 25741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.188:43277: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43277: connect: Connection refused (error 111)
I20260812 02:40:01.103566 25612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } attempt: 2
I20260812 02:40:01.104447 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.104136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43277 (local address 127.0.0.1:35546)
0812 02:40:01.104292 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:01.104360 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 02:40:01.105255 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.104996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43277 (local address 127.0.0.1:35562)
0812 02:40:01.105108 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:01.105184 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 02:40:01.105374 25612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.188:43277: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43277: connect: Connection refused (error 111)
I20260812 02:40:01.125315 25741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } attempt: 2
I20260812 02:40:01.126257 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.125929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43277 (local address 127.0.0.1:35568)
0812 02:40:01.126099 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:01.126169 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260812 02:40:01.126096 25748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:01.126439 25748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:01.126518 25748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 02:40:01.127053 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.126816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43277 (local address 127.0.0.1:35572)
0812 02:40:01.126935 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:01.126987 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 02:40:01.127223 25741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.188:43277: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43277: connect: Connection refused (error 111)
W20260812 02:40:01.131034 25748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:40:01.131110 25748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:01.131130 25748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:01.131146 25748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:40:01.131161 25748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:40:01.137091 25748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:38657,127.3.127.189:44773,127.3.127.188:43277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.188.keytab
--principal=kudu/127.3.127.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.188:43277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.127.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:01.138419 25748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:01.139611 25748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:01.149500 25754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:01.149580 25756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:01.149520 25748 server_base.cc:1034] running on GCE node
W20260812 02:40:01.149530 25761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:01.150419 25748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:01.151767 25748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:01.153013 25748 hybrid_clock.cc:648] HybridClock initialized: now 1786502401152996 us; error 35 us; skew 500 ppm
Aug 12 02:40:01 dist-test-slave-rmhg krb5kdc[25476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502401, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:01.160693 25748 init.cc:377] Logged in from keytab as kudu/127.3.127.188@KRBTEST.COM (short username kudu)
I20260812 02:40:01.161980 25748 webserver.cc:460] Webserver started at http://127.3.127.188:44537/ using document root <none> and password file <none>
I20260812 02:40:01.162650 25748 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:01.162734 25748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:01.162997 25748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:40:01.165369 25748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-2/data/instance:
uuid: "3bd8e01c80c94e06aaec53a74f71d95e"
format_stamp: "Formatted at 2026-08-12 02:40:01 on dist-test-slave-rmhg"
server_key: "3eade31db4538fb2bb45f857008d8553"
server_key_iv: "13e3e2ab3b1b9c919b733b3ea3116ac8"
server_key_version: "encryptionkey@0"
I20260812 02:40:01.165966 25748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-2/wal/instance:
uuid: "3bd8e01c80c94e06aaec53a74f71d95e"
format_stamp: "Formatted at 2026-08-12 02:40:01 on dist-test-slave-rmhg"
server_key: "3eade31db4538fb2bb45f857008d8553"
server_key_iv: "13e3e2ab3b1b9c919b733b3ea3116ac8"
server_key_version: "encryptionkey@0"
I20260812 02:40:01.170245 25748 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:40:01.173307 25780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:01.174889 25748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:40:01.175055 25748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-2/wal
uuid: "3bd8e01c80c94e06aaec53a74f71d95e"
format_stamp: "Formatted at 2026-08-12 02:40:01 on dist-test-slave-rmhg"
server_key: "3eade31db4538fb2bb45f857008d8553"
server_key_iv: "13e3e2ab3b1b9c919b733b3ea3116ac8"
server_key_version: "encryptionkey@0"
I20260812 02:40:01.175189 25748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:01.188112 25748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:01.192289 25748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:01.192499 25748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:01.200141 25748 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.188:43277
I20260812 02:40:01.200183 25866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.188:43277 every 8 connection(s)
I20260812 02:40:01.201366 25748 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-2/data/info.pb
I20260812 02:40:01.204551 25612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } attempt: 3
I20260812 02:40:01.204658 25868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:40:01.210320 25868 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:40:01.210824 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 25748
I20260812 02:40:01.210954 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786502224378336-3582-0/minicluster-data/master-2/wal/instance
I20260812 02:40:01.211196 3582 external_mini_cluster.cc:1428] Setting key 1487c9379e79a598916fd27d2aa7af79
Aug 12 02:40:01 dist-test-slave-rmhg krb5kdc[25476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502400, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for kudu/127.3.127.188@KRBTEST.COM
I20260812 02:40:01.218389 25741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } attempt: 3
Aug 12 02:40:01 dist-test-slave-rmhg krb5kdc[25476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.188@KRBTEST.COM
Aug 12 02:40:01 dist-test-slave-rmhg krb5kdc[25476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502401, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.188@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:40:01.219950 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.205185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43277 (local address 127.0.0.1:35584)
0812 02:40:01.205418 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:01.205434 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:01.205549 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:01.208666 (+ 3117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:01.208671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:01.208930 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:01.209082 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:01.209092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.210558 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.210563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:01.211348 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:01.211365 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.212470 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.212475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:01.212582 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:01.213314 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:01.213331 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:01.214990 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:01.217618 (+ 2628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:01.217622 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:01.217624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:01.217852 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:01.218302 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:01.218305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:01.218307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:01.218423 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:01.219355 (+ 932us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:01.219358 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:01.219496 (+ 138us) client_negotiation.cc:769] Sending connection context
0812 02:40:01.219592 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 02:40:01.219766 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260812 02:40:01.222721 25871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.206063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35584 (local address 127.3.127.188:43277)
0812 02:40:01.206580 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:01.206599 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:01.206640 (+ 41us) server_negotiation.cc:407] Connection header received
0812 02:40:01.207775 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:01.207803 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:01.208248 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:01.208588 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:01.209280 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.210386 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.211512 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.212300 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.212929 (+ 629us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:01.215176 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:01.215182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:01.215193 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:01.215225 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:01.217473 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.217994 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.218003 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.218011 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.218200 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.218556 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.218559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.218561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.219201 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:01.220774 (+ 1573us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:01.221006 (+ 232us) server_negotiation.cc:299] Negotiation successful
0812 02:40:01.221439 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":167,"threads_started":1}
I20260812 02:40:01.225112 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.212663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50970 (local address 127.3.127.190:38657)
0812 02:40:01.212784 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:01.212789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:01.213681 (+ 892us) server_negotiation.cc:407] Connection header received
0812 02:40:01.213821 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:01.213825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:01.213872 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:01.213943 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:01.214992 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.215768 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.216925 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.217293 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.217415 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:01.220083 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:01.220087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:01.220090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:01.220116 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:01.222008 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.222817 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.222821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.222823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.222928 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.223752 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.223757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.223758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.224114 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:01.224221 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:01.224766 (+ 545us) server_negotiation.cc:299] Negotiation successful
0812 02:40:01.224916 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Aug 12 02:40:01 dist-test-slave-rmhg krb5kdc[25476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502400, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.189@KRBTEST.COM for kudu/127.3.127.188@KRBTEST.COM
I20260812 02:40:01.226061 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.212522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:38657 (local address 127.0.0.1:50970)
0812 02:40:01.213530 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:01.213578 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:01.213729 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:01.214135 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:01.214154 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:01.214661 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:01.214852 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:01.214861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.215917 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.215921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:01.216779 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:01.216791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.217439 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.217444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:01.217581 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:01.218235 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:01.218251 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:01.219896 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:01.222416 (+ 2520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:01.222425 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:01.222432 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:01.222709 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:01.223533 (+ 824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:01.223536 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:01.223537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:01.223634 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:01.224278 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:01.224284 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:01.224422 (+ 138us) client_negotiation.cc:769] Sending connection context
0812 02:40:01.225623 (+ 1201us) client_negotiation.cc:240] Negotiation successful
0812 02:40:01.225853 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"thread_start_us":101,"threads_started":1}
I20260812 02:40:01.227396 25868 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:40:01.228755 25612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8: Bootstrap starting.
I20260812 02:40:01.229467 25875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.212290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35598 (local address 127.3.127.188:43277)
0812 02:40:01.212737 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:01.212745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:01.212759 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:40:01.212796 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:01.212799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:01.212851 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:01.212952 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:01.214137 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.214761 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.215972 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.216359 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.216468 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:01.220083 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:01.220087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:01.220090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:01.220115 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:01.223419 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.223937 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.223941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.223943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.224056 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.224376 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.224379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.224381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.224672 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:01.224852 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:01.225746 (+ 894us) server_negotiation.cc:299] Negotiation successful
0812 02:40:01.229282 (+ 3536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3398,"server-negotiator.queue_time_us":366,"thread_start_us":210,"threads_started":1}
I20260812 02:40:01.230545 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.218962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43277 (local address 127.0.0.1:35600)
0812 02:40:01.219159 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:01.219177 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:01.219273 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:01.220092 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:01.220096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:01.220303 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:01.220435 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:01.220444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.221945 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.221949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:01.222574 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:01.222590 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.224138 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.224143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:01.224251 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:01.224761 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:01.224777 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:01.226318 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:01.228506 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:01.228510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:01.228513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:01.228791 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:01.229145 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:01.229175 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:01.229177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:01.229288 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:01.230007 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:01.230011 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:01.230127 (+ 116us) client_negotiation.cc:769] Sending connection context
0812 02:40:01.230217 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 02:40:01.230373 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260812 02:40:01.231484 25878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.219312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35600 (local address 127.3.127.188:43277)
0812 02:40:01.219805 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:01.219815 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:01.219828 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:40:01.219864 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:01.219867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:01.219910 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:01.220558 (+ 648us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:01.220696 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.221810 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.222689 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.223985 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.224583 (+ 598us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:01.226479 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:01.226483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:01.226486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:01.226512 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:01.228409 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.228926 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.228929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.228931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.229045 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.229404 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.229408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.229410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.229882 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:01.231081 (+ 1199us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:01.231162 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 02:40:01.231297 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":407,"spinlock_wait_cycles":768,"thread_start_us":129,"threads_started":1}
I20260812 02:40:01.231961 25612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8: Neither blocks nor log segments found. Creating new log.
I20260812 02:40:01.232942 25612 log.cc:824] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:40:01 dist-test-slave-rmhg krb5kdc[25476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502401, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.188@KRBTEST.COM for kudu/127.3.127.189@KRBTEST.COM
I20260812 02:40:01.235108 25741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c: Bootstrap starting.
I20260812 02:40:01.238026 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.228080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49224 (local address 127.3.127.189:44773)
0812 02:40:01.228204 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:01.228208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:01.228311 (+ 103us) server_negotiation.cc:407] Connection header received
0812 02:40:01.228422 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:01.228425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:01.228476 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:01.228565 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:01.229015 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.229795 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.231321 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.231690 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.231773 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:01.234263 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:01.234267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:01.234270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:01.234299 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:01.235890 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.236494 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.236499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.236501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.236603 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.236945 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.236949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.236950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.237281 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:01.237395 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:01.237771 (+ 376us) server_negotiation.cc:299] Negotiation successful
0812 02:40:01.237890 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 02:40:01.238514 25741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c: Neither blocks nor log segments found. Creating new log.
Aug 12 02:40:01 dist-test-slave-rmhg krb5kdc[25476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:40:01.239569 25741 log.cc:824] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c: Log is configured to *not* fsync() on all Append() calls
I20260812 02:40:01.239643 25612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8: No bootstrap required, opened a new log
I20260812 02:40:01.241694 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.227972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:44773 (local address 127.0.0.1:49224)
0812 02:40:01.228218 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:01.228232 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:01.228335 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:01.228587 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:01.228590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:01.228749 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:01.228865 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:01.228872 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.230215 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.230218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:01.231018 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:01.231032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.231926 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.231929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:01.232024 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:01.232531 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:01.232548 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:01.234096 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:01.236035 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:01.236039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:01.236041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:01.236365 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:01.236731 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:01.236734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:01.236736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:01.236840 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:01.237447 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:01.237450 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:01.237566 (+ 116us) client_negotiation.cc:769] Sending connection context
0812 02:40:01.241279 (+ 3713us) client_negotiation.cc:240] Negotiation successful
0812 02:40:01.241493 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260812 02:40:01.242753 25868 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:40:01.243192 25741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c: No bootstrap required, opened a new log
I20260812 02:40:01.243331 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.231156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50982 (local address 127.3.127.190:38657)
0812 02:40:01.231294 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:01.231299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:01.233416 (+ 2117us) server_negotiation.cc:407] Connection header received
0812 02:40:01.233635 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:01.233638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:01.233685 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:01.233757 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:01.234585 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.235315 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.236364 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.236600 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.236695 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:01.239665 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:01.239670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:01.239672 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:01.239704 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:01.241211 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.241744 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.241748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.241750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.241849 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.242200 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.242204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.242206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.242449 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:01.242618 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:01.243028 (+ 410us) server_negotiation.cc:299] Negotiation successful
0812 02:40:01.243172 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 02:40:01.244423 25612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } } peers { permanent_uuid: "9aaac3eb29e249509b65e72aaaca820c" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } } peers { permanent_uuid: "3bd8e01c80c94e06aaec53a74f71d95e" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } }
I20260812 02:40:01.244735 25612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:40:01.244802 25612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf2561d19a14f498d4eaaccf8aed7f8, State: Initialized, Role: FOLLOWER
I20260812 02:40:01.245440 25612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } } peers { permanent_uuid: "9aaac3eb29e249509b65e72aaaca820c" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } } peers { permanent_uuid: "3bd8e01c80c94e06aaec53a74f71d95e" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } }
I20260812 02:40:01.247076 25612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8 [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 02:40:01 dist-test-slave-rmhg krb5kdc[25476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502401, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.188@KRBTEST.COM for kudu/127.3.127.188@KRBTEST.COM
I20260812 02:40:01.249632 25741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } } peers { permanent_uuid: "9aaac3eb29e249509b65e72aaaca820c" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } } peers { permanent_uuid: "3bd8e01c80c94e06aaec53a74f71d95e" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } }
I20260812 02:40:01.249918 25741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:40:01.249985 25741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aaac3eb29e249509b65e72aaaca820c, State: Initialized, Role: FOLLOWER
I20260812 02:40:01.250643 25741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } } peers { permanent_uuid: "9aaac3eb29e249509b65e72aaaca820c" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } } peers { permanent_uuid: "3bd8e01c80c94e06aaec53a74f71d95e" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } }
Aug 12 02:40:01 dist-test-slave-rmhg krb5kdc[25476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.189@KRBTEST.COM
I20260812 02:40:01.253266 25878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.243771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35612 (local address 127.3.127.188:43277)
0812 02:40:01.243931 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:01.243936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:01.243950 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:40:01.244102 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:01.244105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:01.244156 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:01.244242 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:01.244828 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.245626 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.246631 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.246957 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.247035 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:01.249842 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:01.249847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:01.249849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:01.249877 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:01.251471 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.251944 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.251948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.251949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.252037 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.252322 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.252325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.252327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.252556 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:01.252626 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:01.252939 (+ 313us) server_negotiation.cc:299] Negotiation successful
0812 02:40:01.253078 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 02:40:01.253748 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.243331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43277 (local address 127.0.0.1:35612)
0812 02:40:01.243890 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:01.243903 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:01.243986 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:01.244266 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:01.244268 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:01.244450 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:01.244586 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:01.244593 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.245752 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.245755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:01.246495 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:01.246515 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.247192 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.247195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:01.247272 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:01.247818 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:01.247833 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:01.249695 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:01.251605 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:01.251608 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:01.251610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:01.251842 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:01.252145 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:01.252148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:01.252149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:01.252237 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:01.252683 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:01.252686 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:01.252782 (+ 96us) client_negotiation.cc:769] Sending connection context
0812 02:40:01.253463 (+ 681us) client_negotiation.cc:240] Negotiation successful
0812 02:40:01.253615 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488}
I20260812 02:40:01.252947 25495 master_runner.cc:186] Error getting master registration for 127.3.127.189:44773: OK, instance_id { permanent_uuid: "9aaac3eb29e249509b65e72aaaca820c" instance_seqno: 1786502400960272 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 02:40:01.257759 25868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e: Bootstrap starting.
I20260812 02:40:01.257858 25495 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 02:40:01.260742 25868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e: Neither blocks nor log segments found. Creating new log.
I20260812 02:40:01.262459 25868 log.cc:824] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e: Log is configured to *not* fsync() on all Append() calls
I20260812 02:40:01.265780 25741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:40:01.267274 25868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e: No bootstrap required, opened a new log
W20260812 02:40:01.268692 25900 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:40:01.268877 25900 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:40:01.272871 25868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } } peers { permanent_uuid: "9aaac3eb29e249509b65e72aaaca820c" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } } peers { permanent_uuid: "3bd8e01c80c94e06aaec53a74f71d95e" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } }
I20260812 02:40:01.273258 25868 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:40:01.273334 25868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd8e01c80c94e06aaec53a74f71d95e, State: Initialized, Role: FOLLOWER
I20260812 02:40:01.273952 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.245251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49238 (local address 127.3.127.189:44773)
0812 02:40:01.245367 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:01.245371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:01.245384 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:40:01.245420 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:01.245423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:01.245459 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:01.245541 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:01.246453 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.247197 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.248414 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.248748 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.248827 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:01.256795 (+ 7968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:01.256800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:01.256803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:01.256845 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:01.262794 (+ 5949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.265357 (+ 2563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.265363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.265365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.265516 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.269496 (+ 3980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.269503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.269505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.269819 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:01.270108 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:01.273599 (+ 3491us) server_negotiation.cc:299] Negotiation successful
0812 02:40:01.273786 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 02:40:01.273936 25868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } } peers { permanent_uuid: "9aaac3eb29e249509b65e72aaaca820c" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } } peers { permanent_uuid: "3bd8e01c80c94e06aaec53a74f71d95e" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } }
W20260812 02:40:01.274271 25925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:40:01.274358 25925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:40:01.275035 25882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } } peers { permanent_uuid: "9aaac3eb29e249509b65e72aaaca820c" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } } peers { permanent_uuid: "3bd8e01c80c94e06aaec53a74f71d95e" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } } }
I20260812 02:40:01.275424 25909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } } peers { permanent_uuid: "9aaac3eb29e249509b65e72aaaca820c" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } } peers { permanent_uuid: "3bd8e01c80c94e06aaec53a74f71d95e" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } } }
I20260812 02:40:01.275584 25868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:40:01.275703 25882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:40:01.275820 25909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:40:01.276043 25909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:40:01.276307 25909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } } peers { permanent_uuid: "9aaac3eb29e249509b65e72aaaca820c" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } } peers { permanent_uuid: "3bd8e01c80c94e06aaec53a74f71d95e" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } }
I20260812 02:40:01.277457 25909 leader_election.cc:288] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bf2561d19a14f498d4eaaccf8aed7f8 (127.3.127.190:38657), 3bd8e01c80c94e06aaec53a74f71d95e (127.3.127.188:43277)
I20260812 02:40:01.278439 25824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9aaac3eb29e249509b65e72aaaca820c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bd8e01c80c94e06aaec53a74f71d95e" is_pre_election: true
I20260812 02:40:01.279466 25824 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9aaac3eb29e249509b65e72aaaca820c in term 0.
I20260812 02:40:01.280035 25657 leader_election.cc:302] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bd8e01c80c94e06aaec53a74f71d95e, 9aaac3eb29e249509b65e72aaaca820c; no voters:
W20260812 02:40:01.281576 25945 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:40:01.281680 25945 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:40:01.284222 25559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9aaac3eb29e249509b65e72aaaca820c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8" is_pre_election: true
I20260812 02:40:01.284670 25626 master_runner.cc:263] No leader master found from master 9aaac3eb29e249509b65e72aaaca820c
I20260812 02:40:01.284689 25559 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9aaac3eb29e249509b65e72aaaca820c in term 0.
I20260812 02:40:01.284749 25626 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 02:40:01.285022 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:40:01.285009 25909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:40:01.285135 25909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:40:01.285461 25909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:40:01.288026 25909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } } peers { permanent_uuid: "9aaac3eb29e249509b65e72aaaca820c" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } } peers { permanent_uuid: "3bd8e01c80c94e06aaec53a74f71d95e" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } }
I20260812 02:40:01.291203 25748 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260812 02:40:01.291394 25748 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 02:40:01.292177 25559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9aaac3eb29e249509b65e72aaaca820c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8"
I20260812 02:40:01.292339 25559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:40:01.293830 25559 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9aaac3eb29e249509b65e72aaaca820c in term 1.
I20260812 02:40:01.294299 25658 leader_election.cc:302] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bf2561d19a14f498d4eaaccf8aed7f8, 9aaac3eb29e249509b65e72aaaca820c; no voters:
I20260812 02:40:01.295290 25824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9aaac3eb29e249509b65e72aaaca820c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bd8e01c80c94e06aaec53a74f71d95e"
I20260812 02:40:01.295509 25824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:40:01.296595 25902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:40:01.296895 25902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [term 1 LEADER]: Becoming Leader. State: Replica: 9aaac3eb29e249509b65e72aaaca820c, State: Running, Role: LEADER
I20260812 02:40:01.297381 25902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } } peers { permanent_uuid: "9aaac3eb29e249509b65e72aaaca820c" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } } peers { permanent_uuid: "3bd8e01c80c94e06aaec53a74f71d95e" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } }
I20260812 02:40:01.297861 25824 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9aaac3eb29e249509b65e72aaaca820c in term 1.
I20260812 02:40:01.298979 25909 leader_election.cc:288] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [CANDIDATE]: Term 1 election: Requested vote from peers 1bf2561d19a14f498d4eaaccf8aed7f8 (127.3.127.190:38657), 3bd8e01c80c94e06aaec53a74f71d95e (127.3.127.188:43277)
I20260812 02:40:01.300870 25909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9aaac3eb29e249509b65e72aaaca820c. Latest consensus state: current_term: 1 leader_uuid: "9aaac3eb29e249509b65e72aaaca820c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } } peers { permanent_uuid: "9aaac3eb29e249509b65e72aaaca820c" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } } peers { permanent_uuid: "3bd8e01c80c94e06aaec53a74f71d95e" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } } }
I20260812 02:40:01.301035 25909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [sys.catalog]: This master's current role is: LEADER
I20260812 02:40:01.303615 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.287974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49250 (local address 127.3.127.189:44773)
0812 02:40:01.288229 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:01.288234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:01.290331 (+ 2097us) server_negotiation.cc:407] Connection header received
0812 02:40:01.290553 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:01.290557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:01.290616 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:01.290669 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:01.291956 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.293016 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.294794 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.295104 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.295180 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:01.297619 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:01.297663 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:01.297666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:01.297703 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:01.299655 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.300398 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.300403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.300405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.300512 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.301002 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.301006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.301008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.301599 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:01.301717 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:01.303273 (+ 1556us) server_negotiation.cc:299] Negotiation successful
0812 02:40:01.303420 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260812 02:40:01.306430 25878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.288474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.3.127.188:43277)
0812 02:40:01.288646 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:01.288651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:01.288665 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:40:01.288712 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:01.288715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:01.288765 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:01.288841 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:01.295013 (+ 6172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.295713 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.296796 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.297199 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.297286 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:01.301729 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:01.301734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:01.301736 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:01.301771 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:01.304281 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.304832 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.304836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.304838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.304939 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.305332 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.305335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.305337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.305597 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:01.305723 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:01.306119 (+ 396us) server_negotiation.cc:299] Negotiation successful
0812 02:40:01.306264 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 02:40:01.308706 25965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:40:01.309870 25965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:40:01.313320 25927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } } peers { permanent_uuid: "9aaac3eb29e249509b65e72aaaca820c" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } } peers { permanent_uuid: "3bd8e01c80c94e06aaec53a74f71d95e" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } } }
I20260812 02:40:01.313748 25927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:40:01.316510 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:01.289640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.3.127.190:38657)
0812 02:40:01.289761 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:01.289765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:01.301757 (+ 11992us) server_negotiation.cc:407] Connection header received
0812 02:40:01.301816 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:01.301819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:01.301900 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:01.301978 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:01.306250 (+ 4272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.307998 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.309465 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:01.309812 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:01.310447 (+ 635us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:01.312476 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:01.312480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:01.312482 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:01.312518 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:01.314259 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.314957 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.314961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.314963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.315070 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:01.315406 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:01.315409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:01.315411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:01.315680 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:01.315798 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:01.316196 (+ 398us) server_negotiation.cc:299] Negotiation successful
0812 02:40:01.316345 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 02:40:01.316565 25824 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e [term 1 FOLLOWER]: Refusing update from remote peer 9aaac3eb29e249509b65e72aaaca820c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:40:01.317451 25902 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bd8e01c80c94e06aaec53a74f71d95e" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:40:01.320078 25559 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8 [term 1 FOLLOWER]: Refusing update from remote peer 9aaac3eb29e249509b65e72aaaca820c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:40:01.320926 25909 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:40:01.324543 25927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9aaac3eb29e249509b65e72aaaca820c. Latest consensus state: current_term: 1 leader_uuid: "9aaac3eb29e249509b65e72aaaca820c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } } peers { permanent_uuid: "9aaac3eb29e249509b65e72aaaca820c" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } } peers { permanent_uuid: "3bd8e01c80c94e06aaec53a74f71d95e" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } } }
I20260812 02:40:01.324697 25927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:40:01.328851 25882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9aaac3eb29e249509b65e72aaaca820c. Latest consensus state: current_term: 1 leader_uuid: "9aaac3eb29e249509b65e72aaaca820c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } } peers { permanent_uuid: "9aaac3eb29e249509b65e72aaaca820c" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } } peers { permanent_uuid: "3bd8e01c80c94e06aaec53a74f71d95e" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } } }
I20260812 02:40:01.329027 25882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:40:01.334340 25902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9aaac3eb29e249509b65e72aaaca820c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } } peers { permanent_uuid: "9aaac3eb29e249509b65e72aaaca820c" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } } peers { permanent_uuid: "3bd8e01c80c94e06aaec53a74f71d95e" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } } }
I20260812 02:40:01.334484 25902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [sys.catalog]: This master's current role is: LEADER
I20260812 02:40:01.334607 25966 mvcc.cc:205] Tried to move back new op lower bound from 7317513835782459392 to 7317513835718258688. Current Snapshot: MvccSnapshot[applied={T|T < 7317513835782459392}]
I20260812 02:40:01.334913 25927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9aaac3eb29e249509b65e72aaaca820c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } } peers { permanent_uuid: "9aaac3eb29e249509b65e72aaaca820c" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } } peers { permanent_uuid: "3bd8e01c80c94e06aaec53a74f71d95e" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } } }
I20260812 02:40:01.335048 25927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:40:01.335250 25968 mvcc.cc:205] Tried to move back new op lower bound from 7317513835782459392 to 7317513835718258688. Current Snapshot: MvccSnapshot[applied={T|T < 7317513835782459392}]
I20260812 02:40:01.335902 25965 catalog_manager.cc:1347] Generated new cluster ID: 634c75e8bc004416b37bc42e85f3779e
I20260812 02:40:01.335983 25965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:40:01.341617 25909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9aaac3eb29e249509b65e72aaaca820c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } } peers { permanent_uuid: "9aaac3eb29e249509b65e72aaaca820c" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } } peers { permanent_uuid: "3bd8e01c80c94e06aaec53a74f71d95e" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } } }
I20260812 02:40:01.341790 25909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c [sys.catalog]: This master's current role is: LEADER
I20260812 02:40:01.351794 25882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9aaac3eb29e249509b65e72aaaca820c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf2561d19a14f498d4eaaccf8aed7f8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38657 } } peers { permanent_uuid: "9aaac3eb29e249509b65e72aaaca820c" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 44773 } } peers { permanent_uuid: "3bd8e01c80c94e06aaec53a74f71d95e" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43277 } } }
I20260812 02:40:01.351949 25882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:40:01.352329 25970 mvcc.cc:205] Tried to move back new op lower bound from 7317513835782459392 to 7317513835718258688. Current Snapshot: MvccSnapshot[applied={T|T < 7317513835782459392}]
I20260812 02:40:01.367944 25965 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:40:01.369419 25965 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:40:01.391595 25965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9aaac3eb29e249509b65e72aaaca820c: Generated new TSK 0
I20260812 02:40:01.392628 25965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:40:02.270540 25900 catalog_manager.cc:1259] Loaded cluster ID: 634c75e8bc004416b37bc42e85f3779e
I20260812 02:40:02.270695 25900 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8: loading cluster ID for follower catalog manager: success
I20260812 02:40:02.272537 25900 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8: acquiring CA information for follower catalog manager: success
I20260812 02:40:02.273902 25900 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1bf2561d19a14f498d4eaaccf8aed7f8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:40:02.283105 25945 catalog_manager.cc:1259] Loaded cluster ID: 634c75e8bc004416b37bc42e85f3779e
I20260812 02:40:02.283198 25945 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e: loading cluster ID for follower catalog manager: success
I20260812 02:40:02.285038 25945 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e: acquiring CA information for follower catalog manager: success
I20260812 02:40:02.286466 25945 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3bd8e01c80c94e06aaec53a74f71d95e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:40:02.467516 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 25495
I20260812 02:40:02.485565 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 25626
I20260812 02:40:02.500267 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 25748
2026-08-12T02:40:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3291 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:40:02 dist-test-slave-rmhg krb5kdc[25984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:40:02 dist-test-slave-rmhg krb5kdc[25984](info): set up 2 sockets
Aug 12 02:40:02 dist-test-slave-rmhg krb5kdc[25984](info): commencing operation
krb5kdc: starting...
W20260812 02:40:03.677132 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:40:03 dist-test-slave-rmhg krb5kdc[25984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:40:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:40:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:40:03.810346 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:42561
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:33879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:42561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:40:03.927371 26005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:03.927677 26005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:03.927742 26005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:03.931850 26005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:40:03.931929 26005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:03.931957 26005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:03.931977 26005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:40:03.931998 26005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:40:03.937605 26005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:33879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:42561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:42561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:03.938757 26005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:03.939841 26005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:03.948824 26018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:03.948930 26005 server_base.cc:1034] running on GCE node
W20260812 02:40:03.948834 26013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:03.948817 26016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:03.949782 26005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:03.950999 26005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:03.952189 26005 hybrid_clock.cc:648] HybridClock initialized: now 1786502403952160 us; error 47 us; skew 500 ppm
Aug 12 02:40:03 dist-test-slave-rmhg krb5kdc[25984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502403, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:03.955303 26005 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:40:03.956344 26005 webserver.cc:460] Webserver started at http://127.3.127.190:42145/ using document root <none> and password file <none>
I20260812 02:40:03.956947 26005 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:03.957015 26005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:03.957270 26005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:40:03.959539 26005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "fb89cca73803425c85a5476a254eafb0"
format_stamp: "Formatted at 2026-08-12 02:40:03 on dist-test-slave-rmhg"
server_key: "a3c442aea793093ee379aa232384b64c"
server_key_iv: "837bd037fe4ecc3d50502bf67baae4da"
server_key_version: "encryptionkey@0"
I20260812 02:40:03.960152 26005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "fb89cca73803425c85a5476a254eafb0"
format_stamp: "Formatted at 2026-08-12 02:40:03 on dist-test-slave-rmhg"
server_key: "a3c442aea793093ee379aa232384b64c"
server_key_iv: "837bd037fe4ecc3d50502bf67baae4da"
server_key_version: "encryptionkey@0"
I20260812 02:40:03.964296 26005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 02:40:03.967195 26028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:03.968546 26005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:40:03.968725 26005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "fb89cca73803425c85a5476a254eafb0"
format_stamp: "Formatted at 2026-08-12 02:40:03 on dist-test-slave-rmhg"
server_key: "a3c442aea793093ee379aa232384b64c"
server_key_iv: "837bd037fe4ecc3d50502bf67baae4da"
server_key_version: "encryptionkey@0"
I20260812 02:40:03.968851 26005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:03.987617 26005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:03.991122 26005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:03.991329 26005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:03.998826 26005 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:42561
I20260812 02:40:03.998852 26111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:42561 every 8 connection(s)
I20260812 02:40:03.999915 26005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:40:04.002966 26112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:40:04.008900 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 26005
I20260812 02:40:04.008956 26112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0: Bootstrap starting.
I20260812 02:40:04.009055 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:40:04.009380 3582 external_mini_cluster.cc:1428] Setting key 89ee68848db92314c953800909ae9c66
I20260812 02:40:04.011554 26112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0: Neither blocks nor log segments found. Creating new log.
I20260812 02:40:04.012290 26112 log.cc:824] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0: Log is configured to *not* fsync() on all Append() calls
I20260812 02:40:04.014448 26112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0: No bootstrap required, opened a new log
Aug 12 02:40:04 dist-test-slave-rmhg krb5kdc[25984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:40:04.018155 26112 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb89cca73803425c85a5476a254eafb0" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 42561 } }
I20260812 02:40:04.018424 26112 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:40:04.018482 26112 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb89cca73803425c85a5476a254eafb0, State: Initialized, Role: FOLLOWER
I20260812 02:40:04.018989 26112 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb89cca73803425c85a5476a254eafb0" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 42561 } }
I20260812 02:40:04.019160 26112 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:40:04.019236 26112 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:40:04.019343 26112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:40:04.020571 26112 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb89cca73803425c85a5476a254eafb0" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 42561 } }
I20260812 02:40:04.020979 26112 leader_election.cc:302] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb89cca73803425c85a5476a254eafb0; no voters:
I20260812 02:40:04.021406 26112 leader_election.cc:288] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:40:04.021514 26117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:40:04.021692 26117 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0 [term 1 LEADER]: Becoming Leader. State: Replica: fb89cca73803425c85a5476a254eafb0, State: Running, Role: LEADER
I20260812 02:40:04.022006 26117 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb89cca73803425c85a5476a254eafb0" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 42561 } }
I20260812 02:40:04.022815 26112 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:40:04.023970 26120 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb89cca73803425c85a5476a254eafb0. Latest consensus state: current_term: 1 leader_uuid: "fb89cca73803425c85a5476a254eafb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb89cca73803425c85a5476a254eafb0" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 42561 } } }
I20260812 02:40:04.024104 26120 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0 [sys.catalog]: This master's current role is: LEADER
I20260812 02:40:04.024365 26118 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb89cca73803425c85a5476a254eafb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb89cca73803425c85a5476a254eafb0" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 42561 } } }
I20260812 02:40:04.024441 26118 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0 [sys.catalog]: This master's current role is: LEADER
I20260812 02:40:04.024569 26115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:04.010822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.3.127.190:42561)
0812 02:40:04.011265 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:04.011282 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:04.011320 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:40:04.012005 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:04.012027 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:04.012392 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:04.012727 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:04.013524 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:04.014566 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:04.015445 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:04.015964 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:04.016123 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:04.018824 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:04.018830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:04.018846 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:04.018882 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:04.021060 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:04.021628 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:04.021635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:04.021641 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:04.021815 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:04.022122 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:04.022125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:04.022127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:04.022630 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:04.022836 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:04.023134 (+ 298us) server_negotiation.cc:299] Negotiation successful
0812 02:40:04.023473 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":129,"threads_started":1}
I20260812 02:40:04.024724 26144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:40:04.027971 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:40:04.028034 26144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:40:04.034286 26144 catalog_manager.cc:1347] Generated new cluster ID: 54c9922e12bb4290ac1e73180e59e595
I20260812 02:40:04.034366 26144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:40:04.049255 26144 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:40:04.050352 26144 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:40:04.057662 26144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb89cca73803425c85a5476a254eafb0: Generated new TSK 0
I20260812 02:40:04.058578 26144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:40:04.085115 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 26005
2026-08-12T02:40:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1589 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:40:04 dist-test-slave-rmhg krb5kdc[26167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:40:04 dist-test-slave-rmhg krb5kdc[26167](info): set up 2 sockets
Aug 12 02:40:04 dist-test-slave-rmhg krb5kdc[26167](info): commencing operation
krb5kdc: starting...
W20260812 02:40:05.267446 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:40:05 dist-test-slave-rmhg krb5kdc[26167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:40:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:40:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:40:05.404937 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:36149
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:36149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:40:05.524657 26185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:05.524986 26185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:05.525048 26185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:05.529390 26185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:40:05.529484 26185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:05.529512 26185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:05.529534 26185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:40:05.529554 26185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:40:05.535198 26185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:36149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:36149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:05.536381 26185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:05.537490 26185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:05.546824 26196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:05.546895 26185 server_base.cc:1034] running on GCE node
W20260812 02:40:05.546768 26191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:05.546805 26193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:05.547689 26185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:05.548847 26185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:05.550026 26185 hybrid_clock.cc:648] HybridClock initialized: now 1786502405549996 us; error 43 us; skew 500 ppm
Aug 12 02:40:05 dist-test-slave-rmhg krb5kdc[26167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502405, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:05.552804 26185 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:40:05.553781 26185 webserver.cc:460] Webserver started at http://127.3.127.190:44425/ using document root <none> and password file <none>
I20260812 02:40:05.554348 26185 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:05.554419 26185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:05.554625 26185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:40:05.556790 26185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "e1cc9bb0d8614545889b7fa258668f5c"
format_stamp: "Formatted at 2026-08-12 02:40:05 on dist-test-slave-rmhg"
server_key: "5c5d2c128d8a168b5263bde712972599"
server_key_iv: "94eec46db3daf249ec480b62cb8a0fdf"
server_key_version: "encryptionkey@0"
I20260812 02:40:05.557399 26185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "e1cc9bb0d8614545889b7fa258668f5c"
format_stamp: "Formatted at 2026-08-12 02:40:05 on dist-test-slave-rmhg"
server_key: "5c5d2c128d8a168b5263bde712972599"
server_key_iv: "94eec46db3daf249ec480b62cb8a0fdf"
server_key_version: "encryptionkey@0"
I20260812 02:40:05.561372 26185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:40:05.563987 26207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:05.565346 26185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 02:40:05.565483 26185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "e1cc9bb0d8614545889b7fa258668f5c"
format_stamp: "Formatted at 2026-08-12 02:40:05 on dist-test-slave-rmhg"
server_key: "5c5d2c128d8a168b5263bde712972599"
server_key_iv: "94eec46db3daf249ec480b62cb8a0fdf"
server_key_version: "encryptionkey@0"
I20260812 02:40:05.565600 26185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:05.587617 26185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:05.591221 26185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:05.591446 26185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:05.598968 26185 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:36149
I20260812 02:40:05.599241 26302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:36149 every 8 connection(s)
I20260812 02:40:05.600082 26185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:40:05.603271 26303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:40:05.603396 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 26185
I20260812 02:40:05.603531 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:40:05.603863 3582 external_mini_cluster.cc:1428] Setting key 76770638a7a03ca1784997cd38bd0fb3
I20260812 02:40:05.610316 26303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c: Bootstrap starting.
I20260812 02:40:05.612972 26303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c: Neither blocks nor log segments found. Creating new log.
Aug 12 02:40:05 dist-test-slave-rmhg krb5kdc[26167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:40:05.613873 26303 log.cc:824] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c: Log is configured to *not* fsync() on all Append() calls
I20260812 02:40:05.616226 26303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c: No bootstrap required, opened a new log
I20260812 02:40:05.618553 26310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:05.605484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56718 (local address 127.3.127.190:36149)
0812 02:40:05.605987 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:05.606003 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:05.606043 (+ 40us) server_negotiation.cc:407] Connection header received
0812 02:40:05.606824 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:05.606846 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:05.607207 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:05.607545 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:05.608363 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:05.609448 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:05.610321 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:05.610874 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:05.611039 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:05.613819 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:05.613825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:05.613840 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:05.613874 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:05.615577 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:05.616103 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:05.616107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:05.616112 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:05.616225 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:05.616521 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:05.616524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:05.616525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:05.616946 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:05.617116 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:05.617454 (+ 338us) server_negotiation.cc:299] Negotiation successful
0812 02:40:05.617756 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":158,"threads_started":1}
I20260812 02:40:05.620348 26303 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cc9bb0d8614545889b7fa258668f5c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36149 } }
I20260812 02:40:05.620654 26303 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:40:05.620725 26303 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1cc9bb0d8614545889b7fa258668f5c, State: Initialized, Role: FOLLOWER
I20260812 02:40:05.621230 26303 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cc9bb0d8614545889b7fa258668f5c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36149 } }
I20260812 02:40:05.621372 26303 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:40:05.621423 26303 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:40:05.621497 26303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:40:05.622454 26303 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cc9bb0d8614545889b7fa258668f5c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36149 } }
I20260812 02:40:05.622786 26303 leader_election.cc:302] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1cc9bb0d8614545889b7fa258668f5c; no voters:
I20260812 02:40:05.623098 26303 leader_election.cc:288] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:40:05.623214 26312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:40:05.623409 26312 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 1 LEADER]: Becoming Leader. State: Replica: e1cc9bb0d8614545889b7fa258668f5c, State: Running, Role: LEADER
I20260812 02:40:05.623957 26312 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cc9bb0d8614545889b7fa258668f5c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36149 } }
I20260812 02:40:05.624444 26303 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:40:05.625627 26316 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1cc9bb0d8614545889b7fa258668f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cc9bb0d8614545889b7fa258668f5c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36149 } } }
I20260812 02:40:05.625767 26316 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [sys.catalog]: This master's current role is: LEADER
I20260812 02:40:05.626170 26327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:40:05.626153 26315 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1cc9bb0d8614545889b7fa258668f5c. Latest consensus state: current_term: 1 leader_uuid: "e1cc9bb0d8614545889b7fa258668f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cc9bb0d8614545889b7fa258668f5c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36149 } } }
I20260812 02:40:05.626958 26315 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [sys.catalog]: This master's current role is: LEADER
I20260812 02:40:05.629076 26327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:40:05.634706 26327 catalog_manager.cc:1347] Generated new cluster ID: 101ef9fb4ce645ac94c9bc333173c000
I20260812 02:40:05.634809 26327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:40:05.652048 26327 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:40:05.653079 26327 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:40:05.658723 26327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c: Generated new TSK 0
I20260812 02:40:05.659507 26327 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
I20260812 02:40:05.690135 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:36149
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:40:05.813309 26347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:05.813654 26347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:05.813704 26347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:05.818038 26347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:05.818120 26347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:05.818185 26347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:40:05.823951 26347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:36149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:05.825366 26347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:05.826592 26347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:40:05.837313 26347 server_base.cc:1034] running on GCE node
W20260812 02:40:05.837258 26357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:05.837255 26359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:05.837250 26362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:05.838023 26347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:05.838785 26347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:05.839994 26347 hybrid_clock.cc:648] HybridClock initialized: now 1786502405839974 us; error 32 us; skew 500 ppm
Aug 12 02:40:05 dist-test-slave-rmhg krb5kdc[26167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502405, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:05.843225 26347 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:40:05.844282 26347 webserver.cc:460] Webserver started at http://127.3.127.129:33645/ using document root <none> and password file <none>
I20260812 02:40:05.844920 26347 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:05.845003 26347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:05.845292 26347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:40:05.847662 26347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "9dc6fd13a07b4963b48c6e0c6324f456"
format_stamp: "Formatted at 2026-08-12 02:40:05 on dist-test-slave-rmhg"
server_key: "74ee4b2508ec9bac17f5dc0a65c67cf2"
server_key_iv: "9fd501bec446ab1cd97d47e82cc9d6d6"
server_key_version: "encryptionkey@0"
I20260812 02:40:05.848290 26347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "9dc6fd13a07b4963b48c6e0c6324f456"
format_stamp: "Formatted at 2026-08-12 02:40:05 on dist-test-slave-rmhg"
server_key: "74ee4b2508ec9bac17f5dc0a65c67cf2"
server_key_iv: "9fd501bec446ab1cd97d47e82cc9d6d6"
server_key_version: "encryptionkey@0"
I20260812 02:40:05.852811 26347 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:40:05.855746 26383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:05.857301 26347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 02:40:05.857443 26347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "9dc6fd13a07b4963b48c6e0c6324f456"
format_stamp: "Formatted at 2026-08-12 02:40:05 on dist-test-slave-rmhg"
server_key: "74ee4b2508ec9bac17f5dc0a65c67cf2"
server_key_iv: "9fd501bec446ab1cd97d47e82cc9d6d6"
server_key_version: "encryptionkey@0"
I20260812 02:40:05.857566 26347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:05.883704 26347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:05.887099 26347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:05.887306 26347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:05.887976 26347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:40:05.888891 26347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:40:05.888943 26347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:05.888999 26347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:40:05.889029 26347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:05.898313 26347 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:34143
I20260812 02:40:05.898458 26581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:34143 every 8 connection(s)
I20260812 02:40:05.899364 26347 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:40:05.909085 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 26347
I20260812 02:40:05.909260 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:40:05.909605 3582 external_mini_cluster.cc:1428] Setting key 5ec4610f22c6b1863ddff6204fec56d8
Aug 12 02:40:05 dist-test-slave-rmhg krb5kdc[26167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502405, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:40:05.914011 26310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:05.901390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:49083 (local address 127.3.127.190:36149)
0812 02:40:05.901566 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:05.901571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:05.902531 (+ 960us) server_negotiation.cc:407] Connection header received
0812 02:40:05.903953 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:05.903957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:05.904020 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:05.904096 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:05.905518 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:05.906064 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:05.907265 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:05.907501 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:05.907596 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:05.910344 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:05.910348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:05.910351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:05.910385 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:05.911813 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:05.912352 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:05.912355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:05.912356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:05.912437 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:05.912801 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:05.912804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:05.912806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:05.913004 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:05.913092 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:05.913705 (+ 613us) server_negotiation.cc:299] Negotiation successful
0812 02:40:05.913845 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 02:40:05.914920 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:05.901808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.129:49083)
0812 02:40:05.902346 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:05.902394 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:05.903723 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:05.904302 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:05.904310 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:05.904767 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:05.905353 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:05.905372 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:05.906197 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:05.906199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:05.907132 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:05.907142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:05.907665 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:05.907669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:05.907845 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:05.908439 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:05.908457 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:05.910163 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:05.911934 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:05.911941 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:05.911960 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:05.912241 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:05.912520 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:05.912524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:05.912526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:05.912697 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:05.913109 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:05.913116 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:05.913437 (+ 321us) client_negotiation.cc:769] Sending connection context
0812 02:40:05.913624 (+ 187us) client_negotiation.cc:240] Negotiation successful
0812 02:40:05.914009 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":132,"threads_started":1}
I20260812 02:40:05.916391 26582 heartbeater.cc:347] Connected to a master server at 127.3.127.190:36149
I20260812 02:40:05.916699 26582 heartbeater.cc:464] Registering TS with master...
I20260812 02:40:05.917300 26582 heartbeater.cc:511] Master 127.3.127.190:36149 requested a full tablet report, sending...
I20260812 02:40:05.919036 26242 ts_manager.cc:194] Registered new tserver with Master: 9dc6fd13a07b4963b48c6e0c6324f456 (127.3.127.129:34143)
WARNING: no policy specified for kudu/127.3.127.130@KRBTEST.COM; defaulting to no policy
I20260812 02:40:05.920194 26242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:49083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
I20260812 02:40:05.938460 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:36149
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:40:06.056926 26589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:06.057264 26589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:06.057343 26589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:06.061496 26589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:06.061570 26589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:06.061657 26589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:40:06.067291 26589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:36149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:06.068450 26589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:06.069648 26589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:06.079974 26597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:06.079979 26603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:06.079979 26601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:06.080281 26589 server_base.cc:1034] running on GCE node
I20260812 02:40:06.080837 26589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:06.081588 26589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:06.082774 26589 hybrid_clock.cc:648] HybridClock initialized: now 1786502406082741 us; error 46 us; skew 500 ppm
Aug 12 02:40:06 dist-test-slave-rmhg krb5kdc[26167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502406, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:06.086114 26589 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:40:06.087126 26589 webserver.cc:460] Webserver started at http://127.3.127.130:36211/ using document root <none> and password file <none>
I20260812 02:40:06.087723 26589 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:06.087790 26589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:06.088018 26589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:40:06.090341 26589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "dec61c69893440f7a842a541d868b287"
format_stamp: "Formatted at 2026-08-12 02:40:06 on dist-test-slave-rmhg"
server_key: "0931d971f34d757c47b2fec07ba3e849"
server_key_iv: "649ec7c824f3dad82365965f0ee392bb"
server_key_version: "encryptionkey@0"
I20260812 02:40:06.090952 26589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "dec61c69893440f7a842a541d868b287"
format_stamp: "Formatted at 2026-08-12 02:40:06 on dist-test-slave-rmhg"
server_key: "0931d971f34d757c47b2fec07ba3e849"
server_key_iv: "649ec7c824f3dad82365965f0ee392bb"
server_key_version: "encryptionkey@0"
I20260812 02:40:06.095160 26589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:40:06.098223 26611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:06.099741 26589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:40:06.099890 26589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "dec61c69893440f7a842a541d868b287"
format_stamp: "Formatted at 2026-08-12 02:40:06 on dist-test-slave-rmhg"
server_key: "0931d971f34d757c47b2fec07ba3e849"
server_key_iv: "649ec7c824f3dad82365965f0ee392bb"
server_key_version: "encryptionkey@0"
I20260812 02:40:06.100019 26589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:06.117590 26589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:06.121066 26589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:06.121304 26589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:06.121977 26589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:40:06.122898 26589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:40:06.122949 26589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:06.123006 26589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:40:06.123036 26589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:06.132150 26589 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:34439
I20260812 02:40:06.132184 26808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:34439 every 8 connection(s)
I20260812 02:40:06.133092 26589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:40:06.137003 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 26589
I20260812 02:40:06.137137 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:40:06.137430 3582 external_mini_cluster.cc:1428] Setting key 231bf35bd9675f566d98d4ea5189c263
Aug 12 02:40:06 dist-test-slave-rmhg krb5kdc[26167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502406, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:40:06.148649 26310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:06.135069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:35451 (local address 127.3.127.190:36149)
0812 02:40:06.135202 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:06.135206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:06.136015 (+ 809us) server_negotiation.cc:407] Connection header received
0812 02:40:06.137422 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:06.137426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:06.137498 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:06.137578 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:06.139145 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.139688 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.141366 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.141715 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.141828 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:06.144638 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:06.144643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:06.144646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:06.144681 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:06.146424 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:06.147006 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:06.147010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:06.147012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:06.147123 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:06.147435 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:06.147438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:06.147440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:06.147702 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:06.147808 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:06.148333 (+ 525us) server_negotiation.cc:299] Negotiation successful
0812 02:40:06.148473 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.3.127.131@KRBTEST.COM; defaulting to no policy
I20260812 02:40:06.149555 26815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:06.135372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.130:35451)
0812 02:40:06.135853 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:06.135900 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:06.137192 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:06.137874 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:06.137883 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:06.138351 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:06.138982 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:06.139007 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.139825 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.139829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:06.141200 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:06.141216 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.141864 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.141867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:06.142024 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:06.142669 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:06.142688 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:06.144402 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:06.146579 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:06.146587 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:06.146606 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:06.146892 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:06.147208 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:06.147210 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:06.147211 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:06.147348 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:06.147795 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:06.147800 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:06.148064 (+ 264us) client_negotiation.cc:769] Sending connection context
0812 02:40:06.148248 (+ 184us) client_negotiation.cc:240] Negotiation successful
0812 02:40:06.148538 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":134,"threads_started":1}
I20260812 02:40:06.150822 26811 heartbeater.cc:347] Connected to a master server at 127.3.127.190:36149
I20260812 02:40:06.151094 26811 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.131@KRBTEST.COM" created.
I20260812 02:40:06.151674 26811 heartbeater.cc:511] Master 127.3.127.190:36149 requested a full tablet report, sending...
I20260812 02:40:06.152935 26242 ts_manager.cc:194] Registered new tserver with Master: dec61c69893440f7a842a541d868b287 (127.3.127.130:34439)
I20260812 02:40:06.153698 26242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:35451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
I20260812 02:40:06.168313 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:36149
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:40:06.291541 26818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:06.291859 26818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:06.291904 26818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:06.296185 26818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:06.296274 26818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:06.296337 26818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:40:06.302577 26818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:36149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:06.303860 26818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:06.305068 26818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:06.316298 26828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:06.316337 26829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:06.316522 26818 server_base.cc:1034] running on GCE node
W20260812 02:40:06.316298 26833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:06.317090 26818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:06.317945 26818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:06.319149 26818 hybrid_clock.cc:648] HybridClock initialized: now 1786502406319127 us; error 36 us; skew 500 ppm
Aug 12 02:40:06 dist-test-slave-rmhg krb5kdc[26167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502406, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:06.322433 26818 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:40:06.323580 26818 webserver.cc:460] Webserver started at http://127.3.127.131:39389/ using document root <none> and password file <none>
I20260812 02:40:06.324265 26818 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:06.324347 26818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:06.324643 26818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:40:06.327183 26818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "75b26ee9894d444aa4a42991c9b3f48c"
format_stamp: "Formatted at 2026-08-12 02:40:06 on dist-test-slave-rmhg"
server_key: "aedaa1ba9c78108f3f23314ac5db2519"
server_key_iv: "6fc134a5e5851fffa66be6a093ff4f3c"
server_key_version: "encryptionkey@0"
I20260812 02:40:06.327822 26818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "75b26ee9894d444aa4a42991c9b3f48c"
format_stamp: "Formatted at 2026-08-12 02:40:06 on dist-test-slave-rmhg"
server_key: "aedaa1ba9c78108f3f23314ac5db2519"
server_key_iv: "6fc134a5e5851fffa66be6a093ff4f3c"
server_key_version: "encryptionkey@0"
I20260812 02:40:06.332018 26818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 02:40:06.334731 26844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:06.336227 26818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:40:06.336369 26818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "75b26ee9894d444aa4a42991c9b3f48c"
format_stamp: "Formatted at 2026-08-12 02:40:06 on dist-test-slave-rmhg"
server_key: "aedaa1ba9c78108f3f23314ac5db2519"
server_key_iv: "6fc134a5e5851fffa66be6a093ff4f3c"
server_key_version: "encryptionkey@0"
I20260812 02:40:06.336495 26818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:06.357806 26818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:06.361240 26818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:06.361475 26818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:06.362157 26818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:40:06.363139 26818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:40:06.363193 26818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:06.363251 26818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:40:06.363278 26818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:06.372833 26818 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:38943
I20260812 02:40:06.372908 27037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:38943 every 8 connection(s)
I20260812 02:40:06.373916 26818 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:40:06.376911 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 26818
I20260812 02:40:06.377207 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:40:06.377702 3582 external_mini_cluster.cc:1428] Setting key 84f08b90b6523aa515091b60eff10f33
Aug 12 02:40:06 dist-test-slave-rmhg krb5kdc[26167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502406, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:40:06.391402 26310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:06.376091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:45919 (local address 127.3.127.190:36149)
0812 02:40:06.376351 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:06.376356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:06.377655 (+ 1299us) server_negotiation.cc:407] Connection header received
0812 02:40:06.379445 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:06.379449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:06.379543 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:06.379662 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:06.381828 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.382622 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.384474 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.384719 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.384821 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:06.387616 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:06.387620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:06.387622 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:06.387653 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:06.389137 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:06.389712 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:06.389716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:06.389717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:06.389810 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:06.390188 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:06.390191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:06.390192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:06.390410 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:06.390497 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:06.391088 (+ 591us) server_negotiation.cc:299] Negotiation successful
0812 02:40:06.391233 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260812 02:40:06.392342 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:06.376516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.131:45919)
0812 02:40:06.377439 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:06.377505 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:06.379148 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:06.380114 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:06.380127 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:06.380783 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:06.381634 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:06.381662 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.382786 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.382790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:06.384324 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:06.384338 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.384889 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.384894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:06.385100 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:06.385722 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:06.385740 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:06.387371 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:06.389287 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:06.389292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:06.389308 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:06.389599 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:06.389884 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:06.389886 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:06.389887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:06.390033 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:06.390510 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:06.390516 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:06.390805 (+ 289us) client_negotiation.cc:769] Sending connection context
0812 02:40:06.390989 (+ 184us) client_negotiation.cc:240] Negotiation successful
0812 02:40:06.391385 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":128,"threads_started":1}
I20260812 02:40:06.393613 27038 heartbeater.cc:347] Connected to a master server at 127.3.127.190:36149
I20260812 02:40:06.393873 27038 heartbeater.cc:464] Registering TS with master...
I20260812 02:40:06.394426 27038 heartbeater.cc:511] Master 127.3.127.190:36149 requested a full tablet report, sending...
I20260812 02:40:06.395510 26242 ts_manager.cc:194] Registered new tserver with Master: 75b26ee9894d444aa4a42991c9b3f48c (127.3.127.131:38943)
I20260812 02:40:06.396116 26242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:45919
I20260812 02:40:06.405081 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:40:06.418892 26310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:06.407178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.3.127.190:36149)
0812 02:40:06.407300 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:06.407304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:06.407458 (+ 154us) server_negotiation.cc:407] Connection header received
0812 02:40:06.407591 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:06.407593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:06.407641 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:06.407705 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:06.408555 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.409104 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.410047 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.410277 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.410360 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:06.415969 (+ 5609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:06.415974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:06.415976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:06.416013 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:06.417389 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:06.417824 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:06.417827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:06.417828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:06.417906 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:06.418096 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:06.418098 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:06.418099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:06.418329 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:06.418430 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:06.418650 (+ 220us) server_negotiation.cc:299] Negotiation successful
0812 02:40:06.418762 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 02:40:06.422176 26242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:40:06.425539 26242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:40:06.440622 27058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:06.434851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38306 (local address 127.3.127.129:34143)
0812 02:40:06.435176 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:06.435183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:06.435536 (+ 353us) server_negotiation.cc:407] Connection header received
0812 02:40:06.435751 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:06.435757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:06.435983 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:06.436159 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:40:06.436656 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.437685 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.438999 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.439837 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.439938 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:06.440107 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:06.440334 (+ 227us) server_negotiation.cc:299] Negotiation successful
0812 02:40:06.440445 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":115,"threads_started":1}
I20260812 02:40:06.440639 27055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:06.434719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:34143 (local address 127.0.0.1:38306)
0812 02:40:06.435399 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:06.435438 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:06.435580 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:06.436350 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:06.436357 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:06.436378 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:40:06.436518 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:06.436526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.437790 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.437793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:06.438850 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:06.438860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.439940 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.439943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:06.440118 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:06.440167 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 02:40:06.440284 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 02:40:06.440492 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":69,"threads_started":1}
I20260812 02:40:06.441642 27059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:06.435431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:38943 (local address 127.0.0.1:55688)
0812 02:40:06.436171 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:06.436185 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:06.436268 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:06.436966 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:06.436969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:06.436994 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:40:06.437101 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:06.437108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.438506 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.438510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:06.439737 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:06.439748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.441061 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.441065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:06.441267 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:06.441322 (+ 55us) client_negotiation.cc:769] Sending connection context
0812 02:40:06.441398 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 02:40:06.441499 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":53,"thread_start_us":98,"threads_started":1}
I20260812 02:40:06.441671 27066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:06.435639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55688 (local address 127.3.127.131:38943)
0812 02:40:06.436459 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:06.436475 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:06.436554 (+ 79us) server_negotiation.cc:407] Connection header received
0812 02:40:06.436635 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:06.436641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:06.436830 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:06.436967 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:40:06.437220 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.438361 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.439883 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.440735 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.441055 (+ 320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:06.441242 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:06.441434 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 02:40:06.441528 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":160,"threads_started":1}
I20260812 02:40:06.441850 27067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:06.435864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36432 (local address 127.3.127.130:34439)
0812 02:40:06.436507 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:06.436519 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:06.436542 (+ 23us) server_negotiation.cc:407] Connection header received
0812 02:40:06.436633 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:06.436639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:06.436846 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:06.437004 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:40:06.437430 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.438544 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.439836 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.440649 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.440925 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:06.441092 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:06.441482 (+ 390us) server_negotiation.cc:299] Negotiation successful
0812 02:40:06.441585 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":146,"threads_started":1}
I20260812 02:40:06.442055 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:06.434866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:34439 (local address 127.0.0.1:36432)
0812 02:40:06.436076 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:06.436091 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:06.436201 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:06.437197 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:06.437200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:06.437218 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:40:06.437299 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:06.437304 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.438663 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.438667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:06.439728 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:06.439739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.441225 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.441229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:06.441321 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:06.441340 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 02:40:06.441398 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 02:40:06.441943 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1144,"mutex_wait_us":460,"thread_start_us":144,"threads_started":1}
I20260812 02:40:06.444139 26446 tablet_service.cc:1467] Processing CreateTablet for tablet b0942b9473bf4eadb702b06616203ba6 (DEFAULT_TABLE table=test-table [id=756be3bade5d4b3d9ddde1788bb5c5e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:40:06.444614 26922 tablet_service.cc:1467] Processing CreateTablet for tablet b0942b9473bf4eadb702b06616203ba6 (DEFAULT_TABLE table=test-table [id=756be3bade5d4b3d9ddde1788bb5c5e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:40:06.444689 26698 tablet_service.cc:1467] Processing CreateTablet for tablet b0942b9473bf4eadb702b06616203ba6 (DEFAULT_TABLE table=test-table [id=756be3bade5d4b3d9ddde1788bb5c5e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:40:06.445477 26446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0942b9473bf4eadb702b06616203ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:40:06.445736 26922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0942b9473bf4eadb702b06616203ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:40:06.445802 26698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0942b9473bf4eadb702b06616203ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:40:06.452832 27068 tablet_bootstrap.cc:492] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287: Bootstrap starting.
I20260812 02:40:06.454176 27071 tablet_bootstrap.cc:492] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c: Bootstrap starting.
I20260812 02:40:06.454186 27069 tablet_bootstrap.cc:492] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456: Bootstrap starting.
I20260812 02:40:06.455644 27068 tablet_bootstrap.cc:654] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287: Neither blocks nor log segments found. Creating new log.
I20260812 02:40:06.456427 27068 log.cc:824] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287: Log is configured to *not* fsync() on all Append() calls
I20260812 02:40:06.456537 27071 tablet_bootstrap.cc:654] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c: Neither blocks nor log segments found. Creating new log.
I20260812 02:40:06.456537 27069 tablet_bootstrap.cc:654] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456: Neither blocks nor log segments found. Creating new log.
I20260812 02:40:06.457471 27069 log.cc:824] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456: Log is configured to *not* fsync() on all Append() calls
I20260812 02:40:06.457799 27071 log.cc:824] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c: Log is configured to *not* fsync() on all Append() calls
I20260812 02:40:06.459184 27068 tablet_bootstrap.cc:492] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287: No bootstrap required, opened a new log
I20260812 02:40:06.459393 27068 ts_tablet_manager.cc:1397] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 02:40:06.459748 27069 tablet_bootstrap.cc:492] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456: No bootstrap required, opened a new log
I20260812 02:40:06.459748 27071 tablet_bootstrap.cc:492] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c: No bootstrap required, opened a new log
I20260812 02:40:06.459951 27071 ts_tablet_manager.cc:1397] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 02:40:06.459951 27069 ts_tablet_manager.cc:1397] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 02:40:06.462774 27068 raft_consensus.cc:348] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } } peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } }
I20260812 02:40:06.463042 27068 raft_consensus.cc:374] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:40:06.463096 27068 raft_consensus.cc:729] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec61c69893440f7a842a541d868b287, State: Initialized, Role: FOLLOWER
I20260812 02:40:06.463549 27068 consensus_queue.cc:260] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } } peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } }
I20260812 02:40:06.463990 27071 raft_consensus.cc:348] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } } peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } }
I20260812 02:40:06.464298 27071 raft_consensus.cc:374] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:40:06.464350 27071 raft_consensus.cc:729] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b26ee9894d444aa4a42991c9b3f48c, State: Initialized, Role: FOLLOWER
I20260812 02:40:06.464345 26811 heartbeater.cc:503] Master 127.3.127.190:36149 was elected leader, sending a full tablet report...
I20260812 02:40:06.464543 27069 raft_consensus.cc:348] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } } peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } }
I20260812 02:40:06.464802 27069 raft_consensus.cc:374] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:40:06.464848 27069 raft_consensus.cc:729] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc6fd13a07b4963b48c6e0c6324f456, State: Initialized, Role: FOLLOWER
I20260812 02:40:06.464859 27071 consensus_queue.cc:260] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } } peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } }
I20260812 02:40:06.465342 27069 consensus_queue.cc:260] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } } peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } }
I20260812 02:40:06.465899 27071 ts_tablet_manager.cc:1428] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260812 02:40:06.465909 27068 ts_tablet_manager.cc:1428] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 02:40:06.466298 27069 ts_tablet_manager.cc:1428] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 02:40:06.466295 27038 heartbeater.cc:503] Master 127.3.127.190:36149 was elected leader, sending a full tablet report...
I20260812 02:40:06.466636 26582 heartbeater.cc:503] Master 127.3.127.190:36149 was elected leader, sending a full tablet report...
I20260812 02:40:06.566460 27078 raft_consensus.cc:482] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:40:06.566706 27078 raft_consensus.cc:504] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } } peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } }
I20260812 02:40:06.568004 27078 leader_election.cc:288] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dc6fd13a07b4963b48c6e0c6324f456 (127.3.127.129:34143), dec61c69893440f7a842a541d868b287 (127.3.127.130:34439)
I20260812 02:40:06.572904 27058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:06.568518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:48395 (local address 127.3.127.129:34143)
0812 02:40:06.568673 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:06.568678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:06.569204 (+ 526us) server_negotiation.cc:407] Connection header received
0812 02:40:06.569322 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:06.569325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:06.569381 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:06.569466 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:40:06.569728 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.570369 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.571779 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.572400 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.572507 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:06.572551 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:06.572695 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 02:40:06.572769 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 02:40:06.572904 27067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:06.568421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:57705 (local address 127.3.127.130:34439)
0812 02:40:06.568560 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:06.568565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:06.568583 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:40:06.568672 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:06.568675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:06.568730 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:06.568830 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:40:06.569238 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.570079 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.571653 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.572283 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.572424 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:06.572490 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:06.572680 (+ 190us) server_negotiation.cc:299] Negotiation successful
0812 02:40:06.572769 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 02:40:06.572901 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:06.568308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:34439 (local address 127.3.127.131:57705)
0812 02:40:06.568423 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:06.568439 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:06.568560 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:06.568878 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:06.568882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:06.568909 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:40:06.569046 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:06.569054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.570191 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.570195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:06.571482 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:06.571494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.572417 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.572420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:06.572527 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:06.572557 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 02:40:06.572647 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 02:40:06.572729 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260812 02:40:06.572907 27081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:06.568308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:34143 (local address 127.3.127.131:48395)
0812 02:40:06.569037 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:06.569051 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:06.569216 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:06.569491 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:06.569493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:06.569509 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:40:06.569622 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:06.569627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.570486 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.570490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:06.571644 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:06.571653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.572501 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.572504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:06.572590 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:06.572609 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 02:40:06.572671 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 02:40:06.572729 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":320,"thread_start_us":140,"threads_started":1}
I20260812 02:40:06.573742 26481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0942b9473bf4eadb702b06616203ba6" candidate_uuid: "75b26ee9894d444aa4a42991c9b3f48c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" is_pre_election: true
I20260812 02:40:06.573789 26734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0942b9473bf4eadb702b06616203ba6" candidate_uuid: "75b26ee9894d444aa4a42991c9b3f48c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec61c69893440f7a842a541d868b287" is_pre_election: true
I20260812 02:40:06.574110 26481 raft_consensus.cc:2454] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75b26ee9894d444aa4a42991c9b3f48c in term 0.
I20260812 02:40:06.574110 26734 raft_consensus.cc:2454] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75b26ee9894d444aa4a42991c9b3f48c in term 0.
I20260812 02:40:06.574592 26845 leader_election.cc:302] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75b26ee9894d444aa4a42991c9b3f48c, 9dc6fd13a07b4963b48c6e0c6324f456; no voters:
I20260812 02:40:06.574890 27078 raft_consensus.cc:2781] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:40:06.574991 27078 raft_consensus.cc:482] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:40:06.575042 27078 raft_consensus.cc:3037] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:40:06.576189 27078 raft_consensus.cc:504] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } } peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } }
I20260812 02:40:06.576642 27078 leader_election.cc:288] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c [CANDIDATE]: Term 1 election: Requested vote from peers 9dc6fd13a07b4963b48c6e0c6324f456 (127.3.127.129:34143), dec61c69893440f7a842a541d868b287 (127.3.127.130:34439)
I20260812 02:40:06.577080 26734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0942b9473bf4eadb702b06616203ba6" candidate_uuid: "75b26ee9894d444aa4a42991c9b3f48c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec61c69893440f7a842a541d868b287"
I20260812 02:40:06.577087 26481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0942b9473bf4eadb702b06616203ba6" candidate_uuid: "75b26ee9894d444aa4a42991c9b3f48c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc6fd13a07b4963b48c6e0c6324f456"
I20260812 02:40:06.577265 26734 raft_consensus.cc:3037] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:40:06.577261 26481 raft_consensus.cc:3037] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:40:06.578425 26734 raft_consensus.cc:2454] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b26ee9894d444aa4a42991c9b3f48c in term 1.
I20260812 02:40:06.578617 26481 raft_consensus.cc:2454] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b26ee9894d444aa4a42991c9b3f48c in term 1.
I20260812 02:40:06.578796 26846 leader_election.cc:302] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75b26ee9894d444aa4a42991c9b3f48c, dec61c69893440f7a842a541d868b287; no voters:
I20260812 02:40:06.579013 27078 raft_consensus.cc:2781] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:40:06.579286 27078 raft_consensus.cc:686] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c [term 1 LEADER]: Becoming Leader. State: Replica: 75b26ee9894d444aa4a42991c9b3f48c, State: Running, Role: LEADER
I20260812 02:40:06.579658 27078 consensus_queue.cc:237] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } } peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } }
I20260812 02:40:06.583514 26242 catalog_manager.cc:5495] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c reported cstate change: term changed from 0 to 1, leader changed from <none> to 75b26ee9894d444aa4a42991c9b3f48c (127.3.127.131). New cstate: current_term: 1 leader_uuid: "75b26ee9894d444aa4a42991c9b3f48c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } health_report { overall_health: HEALTHY } } }
I20260812 02:40:06.598253 27066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:06.593975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55702 (local address 127.3.127.131:38943)
0812 02:40:06.594164 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:06.594168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:06.594185 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:40:06.594357 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:06.594361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:06.594424 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:06.594529 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:40:06.594864 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.595378 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.596411 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:06.596715 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:06.596822 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:06.597253 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:40:06.597815 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:40:06.597925 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:06.598029 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 02:40:06.598098 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 02:40:06.605841 26481 raft_consensus.cc:1261] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Refusing update from remote peer 75b26ee9894d444aa4a42991c9b3f48c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:40:06.605863 26734 raft_consensus.cc:1261] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287 [term 1 FOLLOWER]: Refusing update from remote peer 75b26ee9894d444aa4a42991c9b3f48c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:40:06.606760 27082 consensus_queue.cc:1035] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:40:06.606997 27078 consensus_queue.cc:1035] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:40:06.617938 27091 mvcc.cc:205] Tried to move back new op lower bound from 7317513857447534592 to 7317513857353306112. Current Snapshot: MvccSnapshot[applied={T|T < 7317513857447534592}]
I20260812 02:40:06.628795 26238 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:40:06.628993 26238 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:40:06.632179 26238 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c: Sending DeleteTablet for 3 replicas of tablet b0942b9473bf4eadb702b06616203ba6
I20260812 02:40:06.633344 26446 tablet_service.cc:1514] Processing DeleteTablet for tablet b0942b9473bf4eadb702b06616203ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:40:06 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:38306
I20260812 02:40:06.633350 26922 tablet_service.cc:1514] Processing DeleteTablet for tablet b0942b9473bf4eadb702b06616203ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:40:06 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:55688
I20260812 02:40:06.633661 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 26347
I20260812 02:40:06.634320 26698 tablet_service.cc:1514] Processing DeleteTablet for tablet b0942b9473bf4eadb702b06616203ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:40:06 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:36432
I20260812 02:40:06.657603 27109 tablet_replica.cc:330] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c: stopping tablet replica
I20260812 02:40:06.658169 27109 raft_consensus.cc:2229] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:40:06.658713 27109 raft_consensus.cc:2258] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:40:06.661896 27109 ts_tablet_manager.cc:1905] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:40:06.664940 27110 tablet_replica.cc:330] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287: stopping tablet replica
I20260812 02:40:06.665143 27109 ts_tablet_manager.cc:1918] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:40:06.665303 27109 log.cc:1196] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/b0942b9473bf4eadb702b06616203ba6
I20260812 02:40:06.665663 27109 ts_tablet_manager.cc:1939] T b0942b9473bf4eadb702b06616203ba6 P 75b26ee9894d444aa4a42991c9b3f48c: Deleting consensus metadata
I20260812 02:40:06.665699 27110 raft_consensus.cc:2229] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:40:06.666239 27110 raft_consensus.cc:2258] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:40:06.667358 26212 catalog_manager.cc:4841] TS 75b26ee9894d444aa4a42991c9b3f48c (127.3.127.131:38943): tablet b0942b9473bf4eadb702b06616203ba6 (table test-table [id=756be3bade5d4b3d9ddde1788bb5c5e9]) successfully deleted
I20260812 02:40:06.668123 27110 ts_tablet_manager.cc:1905] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 02:40:06.669852 26208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.127.129:34143 (error 108)
I20260812 02:40:06.670195 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 26589
I20260812 02:40:06.697832 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:06.697499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:34143 (local address 127.0.0.1:38314)
0812 02:40:06.697647 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:06.697735 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.129:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 02:40:06.698078 26208 catalog_manager.cc:4584] TS 9dc6fd13a07b4963b48c6e0c6324f456 (127.3.127.129:34143): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0942b9473bf4eadb702b06616203ba6: Network error: Client connection negotiation failed: client connection to 127.3.127.129:34143: connect: Connection refused (error 111)
I20260812 02:40:06.704658 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 26818
I20260812 02:40:06.729454 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:06.704807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:34439 (local address 127.0.0.1:36448)
0812 02:40:06.729216 (+ 24409us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:06.729329 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.130:34439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24228}
I20260812 02:40:06.735811 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 26185
I20260812 02:40:06.758174 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:36149
--webserver_interface=127.3.127.190
--webserver_port=44425
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:36149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:40:06.884665 27113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:06.884949 27113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:06.884989 27113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:06.889240 27113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:40:06.889307 27113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:06.889323 27113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:06.889340 27113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:40:06.889369 27113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:40:06.894963 27113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:36149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:36149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=44425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:06.896167 27113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:06.897333 27113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:06.914330 27126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:06.914315 27134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:06.914332 27130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:06.914546 27113 server_base.cc:1034] running on GCE node
I20260812 02:40:06.915422 27113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:06.916685 27113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:06.917862 27113 hybrid_clock.cc:648] HybridClock initialized: now 1786502406917817 us; error 58 us; skew 500 ppm
Aug 12 02:40:06 dist-test-slave-rmhg krb5kdc[26167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502406, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:06.920878 27113 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:40:06.921954 27113 webserver.cc:460] Webserver started at http://127.3.127.190:44425/ using document root <none> and password file <none>
I20260812 02:40:06.922547 27113 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:06.922623 27113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:06.926831 27113 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:40:06.929132 27145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:06.930527 27113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:40:06.930677 27113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "e1cc9bb0d8614545889b7fa258668f5c"
format_stamp: "Formatted at 2026-08-12 02:40:05 on dist-test-slave-rmhg"
server_key: "5c5d2c128d8a168b5263bde712972599"
server_key_iv: "94eec46db3daf249ec480b62cb8a0fdf"
server_key_version: "encryptionkey@0"
I20260812 02:40:06.931319 27113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:06.946033 27113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:06.949499 27113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:06.949713 27113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:06.957599 27113 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:36149
I20260812 02:40:06.957861 27222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:36149 every 8 connection(s)
I20260812 02:40:06.958847 27113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:40:06.963698 27223 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:40:06.965724 27223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c: Bootstrap starting.
I20260812 02:40:06.966588 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 27113
I20260812 02:40:06.967125 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:34143
--local_ip_for_outbound_sockets=127.3.127.129
--tserver_master_addrs=127.3.127.190:36149
--webserver_port=33645
--webserver_interface=127.3.127.129
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:40:06.985445 27223 log.cc:824] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c: Log is configured to *not* fsync() on all Append() calls
I20260812 02:40:06.994871 27223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:40:06.995281 27223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c: Bootstrap complete.
I20260812 02:40:06.999543 27223 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cc9bb0d8614545889b7fa258668f5c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36149 } }
I20260812 02:40:06.999992 27223 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1cc9bb0d8614545889b7fa258668f5c, State: Initialized, Role: FOLLOWER
I20260812 02:40:07.000723 27223 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cc9bb0d8614545889b7fa258668f5c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36149 } }
I20260812 02:40:07.000892 27223 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:40:07.000990 27223 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:40:07.001097 27223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 1 FOLLOWER]: Advancing to term 2
I20260812 02:40:07.003358 27223 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cc9bb0d8614545889b7fa258668f5c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36149 } }
I20260812 02:40:07.003744 27223 leader_election.cc:302] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1cc9bb0d8614545889b7fa258668f5c; no voters:
I20260812 02:40:07.004112 27223 leader_election.cc:288] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 02:40:07.004237 27229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 2 FOLLOWER]: Leader election won for term 2
I20260812 02:40:07.004452 27229 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 2 LEADER]: Becoming Leader. State: Replica: e1cc9bb0d8614545889b7fa258668f5c, State: Running, Role: LEADER
I20260812 02:40:07.004802 27229 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cc9bb0d8614545889b7fa258668f5c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36149 } }
I20260812 02:40:07.005653 27223 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:40:07.006350 27231 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e1cc9bb0d8614545889b7fa258668f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cc9bb0d8614545889b7fa258668f5c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36149 } } }
I20260812 02:40:07.006515 27231 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [sys.catalog]: This master's current role is: LEADER
I20260812 02:40:07.007252 27232 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1cc9bb0d8614545889b7fa258668f5c. Latest consensus state: current_term: 2 leader_uuid: "e1cc9bb0d8614545889b7fa258668f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cc9bb0d8614545889b7fa258668f5c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36149 } } }
I20260812 02:40:07.007409 27232 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [sys.catalog]: This master's current role is: LEADER
I20260812 02:40:07.007658 27242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:40:07.012987 27242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:40:07.013669 27242 catalog_manager.cc:1259] Loaded cluster ID: 101ef9fb4ce645ac94c9bc333173c000
I20260812 02:40:07.013736 27242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:40:07.015192 27242 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260812 02:40:07.019519 27242 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260812 02:40:07.025774 27242 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:40:07.026710 27242 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c: Loaded TSK: 0
I20260812 02:40:07.028059 27242 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 02:40:07.096361 27226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:07.096657 27226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:07.096704 27226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:07.101102 27226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:07.101217 27226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:07.101292 27226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:40:07.107079 27226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:34143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=33645
--tserver_master_addrs=127.3.127.190:36149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:07.108238 27226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:07.109349 27226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:07.119900 27273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:07.120007 27226 server_base.cc:1034] running on GCE node
W20260812 02:40:07.120252 27279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:07.119900 27282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:07.120767 27226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:07.121554 27226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:07.122738 27226 hybrid_clock.cc:648] HybridClock initialized: now 1786502407122710 us; error 45 us; skew 500 ppm
Aug 12 02:40:07 dist-test-slave-rmhg krb5kdc[26167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:07.125571 27226 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:40:07.126564 27226 webserver.cc:460] Webserver started at http://127.3.127.129:33645/ using document root <none> and password file <none>
I20260812 02:40:07.127126 27226 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:07.127203 27226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:07.131249 27226 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:40:07.133430 27301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:07.134778 27226 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 02:40:07.134900 27226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "9dc6fd13a07b4963b48c6e0c6324f456"
format_stamp: "Formatted at 2026-08-12 02:40:05 on dist-test-slave-rmhg"
server_key: "74ee4b2508ec9bac17f5dc0a65c67cf2"
server_key_iv: "9fd501bec446ab1cd97d47e82cc9d6d6"
server_key_version: "encryptionkey@0"
I20260812 02:40:07.135496 27226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:07.157130 27226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:07.160632 27226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:07.160826 27226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:07.161512 27226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:40:07.162675 27310 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 02:40:07.167181 27226 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 02:40:07.167259 27226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260812 02:40:07.167325 27226 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 02:40:07.169202 27226 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 02:40:07.169260 27226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260812 02:40:07.169553 27310 tablet_bootstrap.cc:492] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456: Bootstrap starting.
I20260812 02:40:07.179589 27226 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:34143
I20260812 02:40:07.179705 27495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:34143 every 8 connection(s)
I20260812 02:40:07.180646 27226 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:40:07.185877 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 27226
I20260812 02:40:07.186376 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:34439
--local_ip_for_outbound_sockets=127.3.127.130
--tserver_master_addrs=127.3.127.190:36149
--webserver_port=36211
--webserver_interface=127.3.127.130
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:40:07.189718 27310 log.cc:824] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:40:07 dist-test-slave-rmhg krb5kdc[26167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:40:07.199275 27502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:07.183041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.129:60177)
0812 02:40:07.183815 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:07.183873 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:07.184762 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:07.186399 (+ 1637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:07.186408 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:07.187201 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:07.187897 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:07.187919 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.189902 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.189906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:07.191119 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:07.191132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.191911 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.191915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:07.192116 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:07.192694 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:07.192712 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:07.194175 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:07.196161 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:07.196167 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:07.196181 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:07.196436 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:07.196785 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:07.196788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:07.196790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:07.196926 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:07.197549 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:07.197556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:07.197805 (+ 249us) client_negotiation.cc:769] Sending connection context
0812 02:40:07.197972 (+ 167us) client_negotiation.cc:240] Negotiation successful
0812 02:40:07.198328 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":138,"threads_started":1}
I20260812 02:40:07.199261 27501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:07.183097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:60177 (local address 127.3.127.190:36149)
0812 02:40:07.183510 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:07.183531 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:07.184026 (+ 495us) server_negotiation.cc:407] Connection header received
0812 02:40:07.185277 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:07.185294 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:07.185825 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:07.186282 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:07.188130 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.189716 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.191255 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.191758 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.191929 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:07.194385 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:07.194390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:07.194404 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:07.194436 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:07.196042 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:07.196553 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:07.196558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:07.196563 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:07.196692 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:07.197017 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:07.197020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:07.197021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:07.197435 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:07.197560 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:07.198120 (+ 560us) server_negotiation.cc:299] Negotiation successful
0812 02:40:07.198393 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":130,"threads_started":1}
I20260812 02:40:07.199602 27310 tablet_bootstrap.cc:492] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:40:07.200071 27310 tablet_bootstrap.cc:492] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456: Bootstrap complete.
I20260812 02:40:07.201714 27310 ts_tablet_manager.cc:1397] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456: Time spent bootstrapping tablet: real 0.032s user 0.015s sys 0.016s
I20260812 02:40:07.202399 27497 heartbeater.cc:347] Connected to a master server at 127.3.127.190:36149
I20260812 02:40:07.202653 27497 heartbeater.cc:464] Registering TS with master...
I20260812 02:40:07.203194 27497 heartbeater.cc:511] Master 127.3.127.190:36149 requested a full tablet report, sending...
I20260812 02:40:07.204857 27310 raft_consensus.cc:348] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } } peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } }
I20260812 02:40:07.205438 27310 raft_consensus.cc:729] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc6fd13a07b4963b48c6e0c6324f456, State: Initialized, Role: FOLLOWER
I20260812 02:40:07.205869 27172 ts_manager.cc:194] Registered new tserver with Master: 9dc6fd13a07b4963b48c6e0c6324f456 (127.3.127.129:34143)
I20260812 02:40:07.205972 27310 consensus_queue.cc:260] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } } peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } }
I20260812 02:40:07.207190 27310 ts_tablet_manager.cc:1428] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 02:40:07.209719 27172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:60177
I20260812 02:40:07.213613 27497 heartbeater.cc:503] Master 127.3.127.190:36149 was elected leader, sending a full tablet report...
Aug 12 02:40:07 dist-test-slave-rmhg krb5kdc[26167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502406, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for kudu/127.3.127.129@KRBTEST.COM
I20260812 02:40:07.223085 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:07.209146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:34143 (local address 127.0.0.1:38322)
0812 02:40:07.209616 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:07.209677 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:07.209831 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:07.211532 (+ 1701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:07.211541 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:07.212327 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:07.212609 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:07.212626 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.214492 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.214496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:07.215496 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:07.215509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.216162 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.216166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:07.216293 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:07.216876 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:07.216892 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:07.218697 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:07.220896 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:07.220925 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:07.220932 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:07.221313 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:07.221693 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:07.221696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:07.221698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:07.221805 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:07.222331 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:07.222337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:07.222488 (+ 151us) client_negotiation.cc:769] Sending connection context
0812 02:40:07.222649 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 02:40:07.222888 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":103,"threads_started":1}
I20260812 02:40:07.223172 27511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:07.209222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38322 (local address 127.3.127.129:34143)
0812 02:40:07.210895 (+ 1673us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:07.210910 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:07.210937 (+ 27us) server_negotiation.cc:407] Connection header received
0812 02:40:07.211020 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:07.211026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:07.211251 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:07.211446 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:07.213260 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.214137 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.215636 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.216032 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.216127 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:07.218896 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:07.218918 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:07.218932 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:07.218962 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:07.220723 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:07.221467 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:07.221475 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:07.221476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:07.221582 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:07.221895 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:07.221897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:07.221898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:07.222220 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:07.222331 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:07.222799 (+ 468us) server_negotiation.cc:299] Negotiation successful
0812 02:40:07.222981 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":832,"thread_start_us":101,"threads_started":1}
I20260812 02:40:07.224346 27375 tablet_service.cc:1514] Processing DeleteTablet for tablet b0942b9473bf4eadb702b06616203ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:40:06 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:38322
I20260812 02:40:07.225309 27512 tablet_replica.cc:330] stopping tablet replica
I20260812 02:40:07.225628 27512 raft_consensus.cc:2229] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:40:07.225883 27512 raft_consensus.cc:2258] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:40:07.226876 27512 ts_tablet_manager.cc:1905] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:40:07.230904 27512 ts_tablet_manager.cc:1918] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:40:07.231024 27512 log.cc:1196] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/wal/wals/b0942b9473bf4eadb702b06616203ba6
I20260812 02:40:07.231279 27512 ts_tablet_manager.cc:1939] T b0942b9473bf4eadb702b06616203ba6 P 9dc6fd13a07b4963b48c6e0c6324f456: Deleting consensus metadata
I20260812 02:40:07.233374 27146 catalog_manager.cc:4841] TS 9dc6fd13a07b4963b48c6e0c6324f456 (127.3.127.129:34143): tablet b0942b9473bf4eadb702b06616203ba6 (table test-table [id=756be3bade5d4b3d9ddde1788bb5c5e9]) successfully deleted
W20260812 02:40:07.319324 27503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:07.319681 27503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:07.319748 27503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:07.324139 27503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:07.324239 27503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:07.324324 27503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:40:07.330271 27503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:34439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=36211
--tserver_master_addrs=127.3.127.190:36149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:07.331534 27503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:07.332669 27503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:07.343921 27519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:07.343921 27520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:07.344421 27503 server_base.cc:1034] running on GCE node
W20260812 02:40:07.343914 27525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:07.345113 27503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:07.345986 27503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:07.347203 27503 hybrid_clock.cc:648] HybridClock initialized: now 1786502407347168 us; error 58 us; skew 500 ppm
Aug 12 02:40:07 dist-test-slave-rmhg krb5kdc[26167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:07.350548 27503 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:40:07.351658 27503 webserver.cc:460] Webserver started at http://127.3.127.130:36211/ using document root <none> and password file <none>
I20260812 02:40:07.352321 27503 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:07.352422 27503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:07.356797 27503 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:40:07.359536 27539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:07.360924 27503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 02:40:07.361091 27503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "dec61c69893440f7a842a541d868b287"
format_stamp: "Formatted at 2026-08-12 02:40:06 on dist-test-slave-rmhg"
server_key: "0931d971f34d757c47b2fec07ba3e849"
server_key_iv: "649ec7c824f3dad82365965f0ee392bb"
server_key_version: "encryptionkey@0"
I20260812 02:40:07.361732 27503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:07.378368 27503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:07.382138 27503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:07.382355 27503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:07.383050 27503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:40:07.384377 27549 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260812 02:40:07.388751 27549 ts_tablet_manager.cc:1852] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260812 02:40:07.389258 27549 ts_tablet_manager.cc:1905] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:40:07.393918 27549 ts_tablet_manager.cc:1918] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:40:07.394068 27549 log.cc:1196] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/b0942b9473bf4eadb702b06616203ba6
I20260812 02:40:07.394410 27549 ts_tablet_manager.cc:1939] T b0942b9473bf4eadb702b06616203ba6 P dec61c69893440f7a842a541d868b287: Deleting consensus metadata
I20260812 02:40:07.394791 27503 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260812 02:40:07.394871 27503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.001s sys 0.000s
I20260812 02:40:07.394944 27503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:40:07.394975 27503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:07.405350 27503 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:34439
I20260812 02:40:07.405551 27738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:34439 every 8 connection(s)
I20260812 02:40:07.406433 27503 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:40:07.406723 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 27503
I20260812 02:40:07.407353 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:38943
--local_ip_for_outbound_sockets=127.3.127.131
--tserver_master_addrs=127.3.127.190:36149
--webserver_port=39389
--webserver_interface=127.3.127.131
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:40:07 dist-test-slave-rmhg krb5kdc[26167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:40:07.423900 27501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:07.409380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:58339 (local address 127.3.127.190:36149)
0812 02:40:07.409561 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:07.409565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:07.410879 (+ 1314us) server_negotiation.cc:407] Connection header received
0812 02:40:07.412129 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:07.412132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:07.412191 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:07.412278 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:07.414208 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.414799 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.416672 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.416950 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.417052 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:07.420219 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:07.420222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:07.420224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:07.420254 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:07.421657 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:07.422230 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:07.422233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:07.422234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:07.422328 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:07.422678 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:07.422681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:07.422682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:07.422882 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:07.422971 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:07.423540 (+ 569us) server_negotiation.cc:299] Negotiation successful
0812 02:40:07.423698 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 02:40:07.424787 27742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:07.410015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.130:58339)
0812 02:40:07.410694 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:07.410745 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:07.411808 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:07.412513 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:07.412524 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:07.413064 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:07.413970 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:07.413997 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.414963 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.414967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:07.416498 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:07.416513 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.417142 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.417147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:07.417421 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:07.418139 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:07.418165 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:07.419990 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:07.421785 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:07.421792 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:07.421813 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:07.422113 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:07.422415 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:07.422418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:07.422420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:07.422575 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:07.422988 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:07.422995 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:07.423285 (+ 290us) client_negotiation.cc:769] Sending connection context
0812 02:40:07.423463 (+ 178us) client_negotiation.cc:240] Negotiation successful
0812 02:40:07.423837 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":184,"threads_started":1}
I20260812 02:40:07.426093 27739 heartbeater.cc:347] Connected to a master server at 127.3.127.190:36149
I20260812 02:40:07.426343 27739 heartbeater.cc:464] Registering TS with master...
I20260812 02:40:07.426867 27739 heartbeater.cc:511] Master 127.3.127.190:36149 requested a full tablet report, sending...
I20260812 02:40:07.427991 27172 ts_manager.cc:194] Registered new tserver with Master: dec61c69893440f7a842a541d868b287 (127.3.127.130:34439)
I20260812 02:40:07.428813 27172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:58339
W20260812 02:40:07.531844 27741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:07.532151 27741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:07.532198 27741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:07.536406 27741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:07.536485 27741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:07.536549 27741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:40:07.542505 27741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:38943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=39389
--tserver_master_addrs=127.3.127.190:36149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:07.543681 27741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:07.544719 27741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:07.555106 27751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:07.555155 27741 server_base.cc:1034] running on GCE node
W20260812 02:40:07.555084 27754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:07.555104 27757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:07.555811 27741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:07.556506 27741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:07.557677 27741 hybrid_clock.cc:648] HybridClock initialized: now 1786502407557654 us; error 36 us; skew 500 ppm
Aug 12 02:40:07 dist-test-slave-rmhg krb5kdc[26167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:07.560627 27741 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:40:07.561658 27741 webserver.cc:460] Webserver started at http://127.3.127.131:39389/ using document root <none> and password file <none>
I20260812 02:40:07.562232 27741 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:07.562312 27741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:07.566433 27741 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:40:07.568683 27769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:07.570150 27741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:40:07.570286 27741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "75b26ee9894d444aa4a42991c9b3f48c"
format_stamp: "Formatted at 2026-08-12 02:40:06 on dist-test-slave-rmhg"
server_key: "aedaa1ba9c78108f3f23314ac5db2519"
server_key_iv: "6fc134a5e5851fffa66be6a093ff4f3c"
server_key_version: "encryptionkey@0"
I20260812 02:40:07.570879 27741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:07.595238 27741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:07.598979 27741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:07.599193 27741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:07.599865 27741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:40:07.600785 27741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:40:07.600838 27741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:07.600894 27741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:40:07.600920 27741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:07.610734 27741 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:38943
I20260812 02:40:07.610762 27982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:38943 every 8 connection(s)
I20260812 02:40:07.611855 27741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:40:07.617616 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 27741
Aug 12 02:40:07 dist-test-slave-rmhg krb5kdc[26167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:40:07.627671 27990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:07.618309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56732 (local address 127.3.127.190:36149)
0812 02:40:07.618571 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:07.618575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:07.618710 (+ 135us) server_negotiation.cc:407] Connection header received
0812 02:40:07.618845 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:07.618847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:07.618887 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:07.618960 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:07.619844 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.620544 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.621663 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.621892 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.621972 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:07.623343 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:07.623346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:07.623348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:07.623387 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:07.624860 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:07.625406 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:07.625409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:07.625410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:07.625490 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:07.626255 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:07.626259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:07.626261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:07.626492 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:07.626742 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:07.627121 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 02:40:07.627280 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":194,"thread_start_us":88,"threads_started":1}
I20260812 02:40:07.628449 27501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:07.613892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:43747 (local address 127.3.127.190:36149)
0812 02:40:07.614024 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:07.614028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:07.615139 (+ 1111us) server_negotiation.cc:407] Connection header received
0812 02:40:07.616056 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:07.616058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:07.616113 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:07.616190 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:07.617577 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.618166 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.619851 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.620236 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.620343 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:07.623158 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:07.623163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:07.623166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:07.623194 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:07.624736 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:07.625367 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:07.625371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:07.625373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:07.625462 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:07.625872 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:07.625875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:07.625877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:07.626128 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:07.626225 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:07.628175 (+ 1950us) server_negotiation.cc:299] Negotiation successful
0812 02:40:07.628302 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":47}
I20260812 02:40:07.629674 27986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:07.614298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.131:43747)
0812 02:40:07.614982 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:07.615030 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:07.615858 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:07.616367 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:07.616374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:07.616814 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:07.617414 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:07.617433 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.618321 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.618324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:07.619750 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:07.619762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.620383 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.620385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:07.620546 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:07.621188 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:07.621209 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:07.622967 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:07.624865 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:07.624872 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:07.624892 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:07.625268 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:07.625555 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:07.625559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:07.625561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:07.625757 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:07.627464 (+ 1707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:07.627472 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:07.627895 (+ 423us) client_negotiation.cc:769] Sending connection context
0812 02:40:07.628091 (+ 196us) client_negotiation.cc:240] Negotiation successful
0812 02:40:07.628491 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":116,"threads_started":1}
I20260812 02:40:07.630776 27983 heartbeater.cc:347] Connected to a master server at 127.3.127.190:36149
I20260812 02:40:07.631016 27983 heartbeater.cc:464] Registering TS with master...
I20260812 02:40:07.631531 27983 heartbeater.cc:511] Master 127.3.127.190:36149 requested a full tablet report, sending...
I20260812 02:40:07.632643 27172 ts_manager.cc:194] Registered new tserver with Master: 75b26ee9894d444aa4a42991c9b3f48c (127.3.127.131:38943)
I20260812 02:40:07.633312 27172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:43747
I20260812 02:40:07.642498 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:40:07.647393 27501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:07.643302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56746 (local address 127.3.127.190:36149)
0812 02:40:07.643459 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:07.643469 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:07.643635 (+ 166us) server_negotiation.cc:407] Connection header received
0812 02:40:07.643834 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:07.643837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:07.643957 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:07.644054 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:40:07.644325 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.644872 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.645871 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.646103 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.646197 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:07.646564 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:40:07.647008 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:40:07.647137 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:07.647180 (+ 43us) server_negotiation.cc:299] Negotiation successful
0812 02:40:07.647241 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 02:40:07.648154 27172 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:40:07.651764 27172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:40:07.660725 27375 tablet_service.cc:1467] Processing CreateTablet for tablet 37db7236ff3e481090b9d97cf48212e7 (DEFAULT_TABLE table=test-table [id=cf9e6f94ecc4438f9a99872a7e4cdeb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:40:07.661558 27375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37db7236ff3e481090b9d97cf48212e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:40:07.664635 27310 tablet_bootstrap.cc:492] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456: Bootstrap starting.
I20260812 02:40:07.665674 27310 tablet_bootstrap.cc:654] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456: Neither blocks nor log segments found. Creating new log.
I20260812 02:40:07.665923 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:07.659662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:34439 (local address 127.0.0.1:36464)
0812 02:40:07.659800 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:07.659817 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:07.659906 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:07.661266 (+ 1360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:07.661269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:07.661290 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:40:07.661414 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:07.661421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.662785 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.662788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:07.663948 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:07.663959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.665437 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.665441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:07.665542 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:07.665594 (+ 52us) client_negotiation.cc:769] Sending connection context
0812 02:40:07.665672 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 02:40:07.665771 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":2}
I20260812 02:40:07.665958 28003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:07.659908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36464 (local address 127.3.127.130:34439)
0812 02:40:07.660741 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:07.660750 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:07.660771 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:40:07.660836 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:07.660840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:07.661066 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:07.661289 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:40:07.661566 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.662651 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.664099 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.665325 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.665435 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:07.665626 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:07.665709 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 02:40:07.665808 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":123,"threads_started":1}
I20260812 02:40:07.666805 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:07.659909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:38943 (local address 127.0.0.1:55710)
0812 02:40:07.660376 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:07.660389 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:07.660491 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:07.661652 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:07.661655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:07.661674 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:40:07.661771 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:07.661777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.663225 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.663229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:07.664240 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:07.664249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.666293 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.666297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:07.666393 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:07.666418 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 02:40:07.666570 (+ 152us) client_negotiation.cc:240] Negotiation successful
0812 02:40:07.666667 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":120,"threads_started":1}
I20260812 02:40:07.666994 28007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:07.660252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55710 (local address 127.3.127.131:38943)
0812 02:40:07.660689 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:07.660702 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:07.660729 (+ 27us) server_negotiation.cc:407] Connection header received
0812 02:40:07.660822 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:07.660829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:07.661057 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:07.661454 (+ 397us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:40:07.661907 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.663070 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.664354 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.665936 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.666068 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:07.666223 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:07.666618 (+ 395us) server_negotiation.cc:299] Negotiation successful
0812 02:40:07.666734 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":193,"threads_started":1}
I20260812 02:40:07.667747 27616 tablet_service.cc:1467] Processing CreateTablet for tablet 37db7236ff3e481090b9d97cf48212e7 (DEFAULT_TABLE table=test-table [id=cf9e6f94ecc4438f9a99872a7e4cdeb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:40:07.668978 27616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37db7236ff3e481090b9d97cf48212e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:40:07.670080 27310 tablet_bootstrap.cc:492] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456: No bootstrap required, opened a new log
I20260812 02:40:07.670202 27310 ts_tablet_manager.cc:1397] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260812 02:40:07.670303 27847 tablet_service.cc:1467] Processing CreateTablet for tablet 37db7236ff3e481090b9d97cf48212e7 (DEFAULT_TABLE table=test-table [id=cf9e6f94ecc4438f9a99872a7e4cdeb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:40:07.670745 27310 raft_consensus.cc:348] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } } peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } }
I20260812 02:40:07.670872 27310 raft_consensus.cc:374] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:40:07.670910 27310 raft_consensus.cc:729] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc6fd13a07b4963b48c6e0c6324f456, State: Initialized, Role: FOLLOWER
I20260812 02:40:07.671060 27310 consensus_queue.cc:260] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } } peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } }
I20260812 02:40:07.671330 27847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37db7236ff3e481090b9d97cf48212e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:40:07.671355 27310 ts_tablet_manager.cc:1428] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:40:07.674563 27549 tablet_bootstrap.cc:492] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287: Bootstrap starting.
I20260812 02:40:07.676642 27549 tablet_bootstrap.cc:654] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287: Neither blocks nor log segments found. Creating new log.
I20260812 02:40:07.677640 27549 log.cc:824] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287: Log is configured to *not* fsync() on all Append() calls
I20260812 02:40:07.679693 27549 tablet_bootstrap.cc:492] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287: No bootstrap required, opened a new log
I20260812 02:40:07.679955 27549 ts_tablet_manager.cc:1397] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 02:40:07.680218 28011 tablet_bootstrap.cc:492] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c: Bootstrap starting.
W20260812 02:40:07.681851 27499 tablet.cc:2369] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:40:07.683333 28011 tablet_bootstrap.cc:654] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c: Neither blocks nor log segments found. Creating new log.
I20260812 02:40:07.684118 27549 raft_consensus.cc:348] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } } peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } }
I20260812 02:40:07.684434 27549 raft_consensus.cc:374] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:40:07.684518 27549 raft_consensus.cc:729] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec61c69893440f7a842a541d868b287, State: Initialized, Role: FOLLOWER
I20260812 02:40:07.684644 28011 log.cc:824] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c: Log is configured to *not* fsync() on all Append() calls
I20260812 02:40:07.685040 27549 consensus_queue.cc:260] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } } peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } }
I20260812 02:40:07.685848 27739 heartbeater.cc:503] Master 127.3.127.190:36149 was elected leader, sending a full tablet report...
I20260812 02:40:07.686173 27549 ts_tablet_manager.cc:1428] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 02:40:07.687176 28011 tablet_bootstrap.cc:492] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c: No bootstrap required, opened a new log
I20260812 02:40:07.687404 28011 ts_tablet_manager.cc:1397] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 02:40:07.691110 28011 raft_consensus.cc:348] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } } peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } }
I20260812 02:40:07.691432 28011 raft_consensus.cc:374] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:40:07.691506 28011 raft_consensus.cc:729] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b26ee9894d444aa4a42991c9b3f48c, State: Initialized, Role: FOLLOWER
I20260812 02:40:07.691951 28011 consensus_queue.cc:260] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } } peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } }
I20260812 02:40:07.692667 27983 heartbeater.cc:503] Master 127.3.127.190:36149 was elected leader, sending a full tablet report...
I20260812 02:40:07.692937 28011 ts_tablet_manager.cc:1428] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260812 02:40:07.839632 28013 raft_consensus.cc:482] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:40:07.839855 28013 raft_consensus.cc:504] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } } peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } }
I20260812 02:40:07.841255 28013 leader_election.cc:288] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75b26ee9894d444aa4a42991c9b3f48c (127.3.127.131:38943), 9dc6fd13a07b4963b48c6e0c6324f456 (127.3.127.129:34143)
I20260812 02:40:07.845393 27742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:07.841507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:38943 (local address 127.3.127.130:39665)
0812 02:40:07.841719 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:07.841735 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:07.841846 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:07.842207 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:07.842210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:07.842235 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:40:07.842377 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:07.842384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.843223 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.843227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:07.844113 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:07.844123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.844885 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.844889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:07.845002 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:07.845031 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 02:40:07.845102 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 02:40:07.845236 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":25}
I20260812 02:40:07.845392 28007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:07.841620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:39665 (local address 127.3.127.131:38943)
0812 02:40:07.841757 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:07.841761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:07.841780 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:40:07.841979 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:07.841984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:07.842047 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:07.842125 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:40:07.842493 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.843102 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.844264 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.844755 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.844860 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:07.844905 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:07.845141 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 02:40:07.845258 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 02:40:07.845600 28016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:07.841654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:46141 (local address 127.3.127.129:34143)
0812 02:40:07.841958 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:07.841962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:07.842144 (+ 182us) server_negotiation.cc:407] Connection header received
0812 02:40:07.842233 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:07.842236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:07.842303 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:07.842432 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:40:07.842722 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.843260 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.844430 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.844946 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.845077 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:07.845287 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:07.845340 (+ 53us) server_negotiation.cc:299] Negotiation successful
0812 02:40:07.845411 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":81,"threads_started":1}
I20260812 02:40:07.846097 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:07.841576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:34143 (local address 127.3.127.130:46141)
0812 02:40:07.841991 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:07.842006 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:07.842144 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:07.842448 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:07.842451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:07.842471 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:40:07.842590 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:07.842597 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.843398 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.843401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:07.844296 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:07.844306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.845080 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.845084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:07.845221 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:07.845245 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 02:40:07.845303 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 02:40:07.845367 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":24,"thread_start_us":119,"threads_started":1}
I20260812 02:40:07.846282 27897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37db7236ff3e481090b9d97cf48212e7" candidate_uuid: "dec61c69893440f7a842a541d868b287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b26ee9894d444aa4a42991c9b3f48c" is_pre_election: true
I20260812 02:40:07.846679 27897 raft_consensus.cc:2454] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec61c69893440f7a842a541d868b287 in term 0.
I20260812 02:40:07.846855 27410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37db7236ff3e481090b9d97cf48212e7" candidate_uuid: "dec61c69893440f7a842a541d868b287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" is_pre_election: true
I20260812 02:40:07.847208 27410 raft_consensus.cc:2454] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec61c69893440f7a842a541d868b287 in term 0.
I20260812 02:40:07.847216 27541 leader_election.cc:302] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75b26ee9894d444aa4a42991c9b3f48c, dec61c69893440f7a842a541d868b287; no voters:
I20260812 02:40:07.847553 28013 raft_consensus.cc:2781] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:40:07.847673 28013 raft_consensus.cc:482] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:40:07.847729 28013 raft_consensus.cc:3037] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:40:07.848790 28013 raft_consensus.cc:504] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } } peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } }
I20260812 02:40:07.849145 28013 leader_election.cc:288] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 [CANDIDATE]: Term 1 election: Requested vote from peers 75b26ee9894d444aa4a42991c9b3f48c (127.3.127.131:38943), 9dc6fd13a07b4963b48c6e0c6324f456 (127.3.127.129:34143)
I20260812 02:40:07.849570 27410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37db7236ff3e481090b9d97cf48212e7" candidate_uuid: "dec61c69893440f7a842a541d868b287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc6fd13a07b4963b48c6e0c6324f456"
I20260812 02:40:07.849709 27897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37db7236ff3e481090b9d97cf48212e7" candidate_uuid: "dec61c69893440f7a842a541d868b287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b26ee9894d444aa4a42991c9b3f48c"
I20260812 02:40:07.849721 27410 raft_consensus.cc:3037] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:40:07.849840 27897 raft_consensus.cc:3037] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:40:07.851009 27410 raft_consensus.cc:2454] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec61c69893440f7a842a541d868b287 in term 1.
I20260812 02:40:07.851009 27897 raft_consensus.cc:2454] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec61c69893440f7a842a541d868b287 in term 1.
I20260812 02:40:07.851320 27541 leader_election.cc:302] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 75b26ee9894d444aa4a42991c9b3f48c, 9dc6fd13a07b4963b48c6e0c6324f456, dec61c69893440f7a842a541d868b287; no voters:
I20260812 02:40:07.851516 28013 raft_consensus.cc:2781] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:40:07.851737 28013 raft_consensus.cc:686] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 [term 1 LEADER]: Becoming Leader. State: Replica: dec61c69893440f7a842a541d868b287, State: Running, Role: LEADER
I20260812 02:40:07.852101 28013 consensus_queue.cc:237] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } } peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } }
I20260812 02:40:07.855471 27171 catalog_manager.cc:5495] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 reported cstate change: term changed from 0 to 1, leader changed from <none> to dec61c69893440f7a842a541d868b287 (127.3.127.130). New cstate: current_term: 1 leader_uuid: "dec61c69893440f7a842a541d868b287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } health_report { overall_health: UNKNOWN } } }
W20260812 02:40:07.862826 27985 tablet.cc:2369] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:40:07.905390 28003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:07.901673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36470 (local address 127.3.127.130:34439)
0812 02:40:07.901800 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:07.901804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:07.901819 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:40:07.901906 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:07.901908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:07.901969 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:07.902067 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:40:07.902368 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.902884 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.903727 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:07.903961 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:07.904066 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:07.904424 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:40:07.904891 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:40:07.905015 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:07.905183 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 02:40:07.905251 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260812 02:40:07.907831 27740 tablet.cc:2369] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:40:07.911929 27410 raft_consensus.cc:1261] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Refusing update from remote peer dec61c69893440f7a842a541d868b287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:40:07.911926 27897 raft_consensus.cc:1261] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c [term 1 FOLLOWER]: Refusing update from remote peer dec61c69893440f7a842a541d868b287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:40:07.912665 28017 consensus_queue.cc:1035] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:40:07.912859 28013 consensus_queue.cc:1035] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:40:07.920516 28029 mvcc.cc:205] Tried to move back new op lower bound from 7317513862799679488 to 7317513862564896768. Current Snapshot: MvccSnapshot[applied={T|T < 7317513862799679488}]
I20260812 02:40:07.920910 28026 mvcc.cc:205] Tried to move back new op lower bound from 7317513862799679488 to 7317513862564896768. Current Snapshot: MvccSnapshot[applied={T|T < 7317513862799679488}]
I20260812 02:40:07.929450 27171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:40:07.929670 27171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:40:07.932353 27171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c: Sending DeleteTablet for 3 replicas of tablet 37db7236ff3e481090b9d97cf48212e7
I20260812 02:40:07.933226 27375 tablet_service.cc:1514] Processing DeleteTablet for tablet 37db7236ff3e481090b9d97cf48212e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:40:07 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:38322
I20260812 02:40:07.933362 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 27226
I20260812 02:40:07.933687 27616 tablet_service.cc:1514] Processing DeleteTablet for tablet 37db7236ff3e481090b9d97cf48212e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:40:07 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:36464
I20260812 02:40:07.936292 27847 tablet_service.cc:1514] Processing DeleteTablet for tablet 37db7236ff3e481090b9d97cf48212e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:40:07 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:55710
I20260812 02:40:07.947470 28043 tablet_replica.cc:330] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c: stopping tablet replica
I20260812 02:40:07.947932 28043 raft_consensus.cc:2229] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:40:07.948390 28043 raft_consensus.cc:2258] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:40:07.952452 28041 tablet_replica.cc:330] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287: stopping tablet replica
I20260812 02:40:07.952912 28041 raft_consensus.cc:2229] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:40:07.953482 28041 raft_consensus.cc:2258] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:40:07.954067 28043 ts_tablet_manager.cc:1905] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:40:07.955188 28041 ts_tablet_manager.cc:1905] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:40:07.957955 28041 ts_tablet_manager.cc:1918] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:40:07.958060 28041 log.cc:1196] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/37db7236ff3e481090b9d97cf48212e7
I20260812 02:40:07.958310 28041 ts_tablet_manager.cc:1939] T 37db7236ff3e481090b9d97cf48212e7 P dec61c69893440f7a842a541d868b287: Deleting consensus metadata
I20260812 02:40:07.959647 27147 catalog_manager.cc:4841] TS dec61c69893440f7a842a541d868b287 (127.3.127.130:34439): tablet 37db7236ff3e481090b9d97cf48212e7 (table test-table [id=cf9e6f94ecc4438f9a99872a7e4cdeb8]) successfully deleted
I20260812 02:40:07.959919 28043 ts_tablet_manager.cc:1918] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:40:07.960091 28043 log.cc:1196] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/37db7236ff3e481090b9d97cf48212e7
I20260812 02:40:07.960480 28043 ts_tablet_manager.cc:1939] T 37db7236ff3e481090b9d97cf48212e7 P 75b26ee9894d444aa4a42991c9b3f48c: Deleting consensus metadata
W20260812 02:40:07.960796 27146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.127.129:34143 (error 108)
I20260812 02:40:07.962150 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 27503
I20260812 02:40:07.962739 27147 catalog_manager.cc:4841] TS 75b26ee9894d444aa4a42991c9b3f48c (127.3.127.131:38943): tablet 37db7236ff3e481090b9d97cf48212e7 (table test-table [id=cf9e6f94ecc4438f9a99872a7e4cdeb8]) successfully deleted
I20260812 02:40:07.970795 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:07.970409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:34143 (local address 127.0.0.1:38336)
0812 02:40:07.970625 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:07.970703 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.129:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260812 02:40:07.970997 27146 catalog_manager.cc:4584] TS 9dc6fd13a07b4963b48c6e0c6324f456 (127.3.127.129:34143): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 37db7236ff3e481090b9d97cf48212e7: Network error: Client connection negotiation failed: client connection to 127.3.127.129:34143: connect: Connection refused (error 111)
I20260812 02:40:07.978619 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 27741
I20260812 02:40:07.994043 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 27113
I20260812 02:40:08.024714 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:36149
--webserver_interface=127.3.127.190
--webserver_port=44425
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:36149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:40:08.143527 28048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:08.143850 28048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:08.143893 28048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:08.148052 28048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:40:08.148133 28048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:08.148154 28048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:08.148168 28048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:40:08.148182 28048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:40:08.154069 28048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:36149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:36149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=44425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:08.155283 28048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:08.156368 28048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:08.165633 28053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:08.165606 28056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:08.165640 28058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:08.165884 28048 server_base.cc:1034] running on GCE node
I20260812 02:40:08.166620 28048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:08.167889 28048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:08.169080 28048 hybrid_clock.cc:648] HybridClock initialized: now 1786502408169055 us; error 41 us; skew 500 ppm
Aug 12 02:40:08 dist-test-slave-rmhg krb5kdc[26167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502408, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:08.172133 28048 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:40:08.173146 28048 webserver.cc:460] Webserver started at http://127.3.127.190:44425/ using document root <none> and password file <none>
I20260812 02:40:08.173785 28048 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:08.173851 28048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:08.178277 28048 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:40:08.180800 28071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:08.182368 28048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:40:08.182507 28048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "e1cc9bb0d8614545889b7fa258668f5c"
format_stamp: "Formatted at 2026-08-12 02:40:05 on dist-test-slave-rmhg"
server_key: "5c5d2c128d8a168b5263bde712972599"
server_key_iv: "94eec46db3daf249ec480b62cb8a0fdf"
server_key_version: "encryptionkey@0"
I20260812 02:40:08.183147 28048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:08.201782 28048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:08.205542 28048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:08.205763 28048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:08.213505 28048 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:36149
I20260812 02:40:08.213819 28183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:36149 every 8 connection(s)
I20260812 02:40:08.215209 28048 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:40:08.220741 28184 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:40:08.222577 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 28048
I20260812 02:40:08.223032 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:34143
--local_ip_for_outbound_sockets=127.3.127.129
--tserver_master_addrs=127.3.127.190:36149
--webserver_port=33645
--webserver_interface=127.3.127.129
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:40:08.223052 28184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c: Bootstrap starting.
I20260812 02:40:08.243438 28184 log.cc:824] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c: Log is configured to *not* fsync() on all Append() calls
I20260812 02:40:08.256049 28184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:40:08.256475 28184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c: Bootstrap complete.
I20260812 02:40:08.260895 28184 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cc9bb0d8614545889b7fa258668f5c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36149 } }
I20260812 02:40:08.261411 28184 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1cc9bb0d8614545889b7fa258668f5c, State: Initialized, Role: FOLLOWER
I20260812 02:40:08.261978 28184 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cc9bb0d8614545889b7fa258668f5c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36149 } }
I20260812 02:40:08.262135 28184 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:40:08.262207 28184 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:40:08.262310 28184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 2 FOLLOWER]: Advancing to term 3
I20260812 02:40:08.264480 28184 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cc9bb0d8614545889b7fa258668f5c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36149 } }
I20260812 02:40:08.264853 28184 leader_election.cc:302] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1cc9bb0d8614545889b7fa258668f5c; no voters:
I20260812 02:40:08.265261 28184 leader_election.cc:288] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 02:40:08.265367 28192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 3 FOLLOWER]: Leader election won for term 3
I20260812 02:40:08.265581 28192 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [term 3 LEADER]: Becoming Leader. State: Replica: e1cc9bb0d8614545889b7fa258668f5c, State: Running, Role: LEADER
I20260812 02:40:08.265925 28192 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cc9bb0d8614545889b7fa258668f5c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36149 } }
I20260812 02:40:08.266672 28184 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:40:08.267438 28194 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e1cc9bb0d8614545889b7fa258668f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cc9bb0d8614545889b7fa258668f5c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36149 } } }
I20260812 02:40:08.267585 28194 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [sys.catalog]: This master's current role is: LEADER
I20260812 02:40:08.268023 28196 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1cc9bb0d8614545889b7fa258668f5c. Latest consensus state: current_term: 3 leader_uuid: "e1cc9bb0d8614545889b7fa258668f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cc9bb0d8614545889b7fa258668f5c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36149 } } }
I20260812 02:40:08.268172 28196 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cc9bb0d8614545889b7fa258668f5c [sys.catalog]: This master's current role is: LEADER
I20260812 02:40:08.268759 28205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:40:08.273303 28205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:40:08.273923 28205 catalog_manager.cc:1259] Loaded cluster ID: 101ef9fb4ce645ac94c9bc333173c000
I20260812 02:40:08.273980 28205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260812 02:40:08.274910 28205 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f194dc949ad google::LogMessage::Fail() at ??:0
@ 0x7f194dc9788a google::LogMessage::SendToLog() at ??:0
@ 0x7f194dc944de google::LogMessage::Flush() at ??:0
@ 0x7f194dc95a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f19585d3fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f19585d4d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f19585d1579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f1958615640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f194fee63d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f194fee6c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f194fee83f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f194fed7512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f195151b6db start_thread at ??:0
@ 0x7f194c79261f clone at ??:0
W20260812 02:40:08.362927 28188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:08.363224 28188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:08.363291 28188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:08.367465 28188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:08.367542 28188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:08.367625 28188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:40:08.373212 28188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:34143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=33645
--tserver_master_addrs=127.3.127.190:36149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:08.374385 28188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:08.375478 28188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:08.388696 28250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:08.388742 28249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:08.388921 28255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:08.389567 28188 server_base.cc:1034] running on GCE node
I20260812 02:40:08.390260 28188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:08.391142 28188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:08.394425 28188 hybrid_clock.cc:648] HybridClock initialized: now 1786502408394381 us; error 49 us; skew 500 ppm
Aug 12 02:40:08 dist-test-slave-rmhg krb5kdc[26167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502408, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:08.397516 28188 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:40:08.398598 28188 webserver.cc:460] Webserver started at http://127.3.127.129:33645/ using document root <none> and password file <none>
I20260812 02:40:08.399194 28188 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:08.399268 28188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:08.403733 28188 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:40:08.406199 28264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:08.407721 28188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:40:08.407852 28188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "9dc6fd13a07b4963b48c6e0c6324f456"
format_stamp: "Formatted at 2026-08-12 02:40:05 on dist-test-slave-rmhg"
server_key: "74ee4b2508ec9bac17f5dc0a65c67cf2"
server_key_iv: "9fd501bec446ab1cd97d47e82cc9d6d6"
server_key_version: "encryptionkey@0"
I20260812 02:40:08.408502 28188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:08.422438 28188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:08.425858 28188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:08.426054 28188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:08.426674 28188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:40:08.427816 28278 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 02:40:08.433862 28188 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 02:40:08.433959 28188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260812 02:40:08.434037 28188 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 02:40:08.435920 28188 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 02:40:08.435977 28188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260812 02:40:08.436339 28278 tablet_bootstrap.cc:492] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456: Bootstrap starting.
I20260812 02:40:08.446919 28188 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:34143
I20260812 02:40:08.447270 28454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:34143 every 8 connection(s)
I20260812 02:40:08.448431 28188 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:40:08.448977 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 28188
I20260812 02:40:08.449518 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:34439
--local_ip_for_outbound_sockets=127.3.127.130
--tserver_master_addrs=127.3.127.190:36149
--webserver_port=36211
--webserver_interface=127.3.127.130
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:40:08.459364 28278 log.cc:824] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456: Log is configured to *not* fsync() on all Append() calls
I20260812 02:40:08.467087 28278 tablet_bootstrap.cc:492] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:40:08.467506 28278 tablet_bootstrap.cc:492] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456: Bootstrap complete.
I20260812 02:40:08.468621 28278 ts_tablet_manager.cc:1397] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456: Time spent bootstrapping tablet: real 0.033s user 0.015s sys 0.012s
I20260812 02:40:08.471292 28278 raft_consensus.cc:348] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } } peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } }
I20260812 02:40:08.471750 28278 raft_consensus.cc:729] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc6fd13a07b4963b48c6e0c6324f456, State: Initialized, Role: FOLLOWER
I20260812 02:40:08.472280 28278 consensus_queue.cc:260] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } } peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } }
I20260812 02:40:08.473325 28278 ts_tablet_manager.cc:1428] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260812 02:40:08.522327 28462 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:40:08.451960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.129:49337)
0812 02:40:08.452503 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:08.452565 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:08.453734 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:08.521460 (+ 67726us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":147,"threads_started":1}
W20260812 02:40:08.522981 28459 heartbeater.cc:644] Failed to heartbeat to 127.3.127.190:36149 (0 consecutive failures): Network error: Failed to ping master at 127.3.127.190:36149: Client connection negotiation failed: client connection to 127.3.127.190:36149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 02:40:08.523849 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:08.523530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.129:53753)
0812 02:40:08.523705 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:08.523780 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260812 02:40:08.524662 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:08.524393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.129:34227)
0812 02:40:08.524526 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:08.524575 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 02:40:08.524847 28459 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 02:40:08.587859 28461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:08.588140 28461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:08.588186 28461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:08.592640 28461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:08.592736 28461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:08.592803 28461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:40:08.598526 28461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:34439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=36211
--tserver_master_addrs=127.3.127.190:36149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:08.599705 28461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:08.600772 28461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:08.610893 28478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:08.610911 28480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:08.611035 28482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:08.611333 28461 server_base.cc:1034] running on GCE node
I20260812 02:40:08.611799 28461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:08.612506 28461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:08.613687 28461 hybrid_clock.cc:648] HybridClock initialized: now 1786502408613653 us; error 47 us; skew 500 ppm
Aug 12 02:40:08 dist-test-slave-rmhg krb5kdc[26167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502408, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:08.616704 28461 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:40:08.617810 28461 webserver.cc:460] Webserver started at http://127.3.127.130:36211/ using document root <none> and password file <none>
I20260812 02:40:08.618420 28461 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:08.618496 28461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:08.622972 28461 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:40:08.625560 28496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:08.627187 28461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 02:40:08.627370 28461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "dec61c69893440f7a842a541d868b287"
format_stamp: "Formatted at 2026-08-12 02:40:06 on dist-test-slave-rmhg"
server_key: "0931d971f34d757c47b2fec07ba3e849"
server_key_iv: "649ec7c824f3dad82365965f0ee392bb"
server_key_version: "encryptionkey@0"
I20260812 02:40:08.628064 28461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:08.648777 28461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:08.652213 28461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:08.652391 28461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:08.653026 28461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:40:08.654009 28461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:40:08.654062 28461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:08.654105 28461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:40:08.654119 28461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:08.663650 28461 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:34439
I20260812 02:40:08.663674 28692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:34439 every 8 connection(s)
I20260812 02:40:08.664770 28461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:40:08.669258 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 28461
I20260812 02:40:08.669384 28698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:08.667162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.130:58219)
0812 02:40:08.667696 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:08.667824 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":133,"threads_started":1}
I20260812 02:40:08.669785 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:38943
--local_ip_for_outbound_sockets=127.3.127.131
--tserver_master_addrs=127.3.127.190:36149
--webserver_port=39389
--webserver_interface=127.3.127.131
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:40:08.670015 28696 heartbeater.cc:644] Failed to heartbeat to 127.3.127.190:36149 (0 consecutive failures): Network error: Failed to ping master at 127.3.127.190:36149: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
I20260812 02:40:08.671135 28698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:08.670811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.130:36865)
0812 02:40:08.670971 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:08.671044 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 02:40:08.672060 28698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:08.671767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.130:54135)
0812 02:40:08.671931 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:08.671991 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260812 02:40:08.672317 28696 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 02:40:08.791894 28699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:08.792215 28699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:08.792268 28699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:08.796583 28699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:08.796679 28699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:08.796742 28699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:40:08.802546 28699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:38943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=39389
--tserver_master_addrs=127.3.127.190:36149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:08.803793 28699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:08.804976 28699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:08.816195 28719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:08.816205 28714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:08.816319 28699 server_base.cc:1034] running on GCE node
W20260812 02:40:08.816205 28715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:08.816992 28699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:08.817865 28699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:08.819046 28699 hybrid_clock.cc:648] HybridClock initialized: now 1786502408819017 us; error 44 us; skew 500 ppm
Aug 12 02:40:08 dist-test-slave-rmhg krb5kdc[26167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502408, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:08.822163 28699 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:40:08.823163 28699 webserver.cc:460] Webserver started at http://127.3.127.131:39389/ using document root <none> and password file <none>
I20260812 02:40:08.823761 28699 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:08.823838 28699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:08.828190 28699 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:40:08.830717 28731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:08.832305 28699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:40:08.832446 28699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "75b26ee9894d444aa4a42991c9b3f48c"
format_stamp: "Formatted at 2026-08-12 02:40:06 on dist-test-slave-rmhg"
server_key: "aedaa1ba9c78108f3f23314ac5db2519"
server_key_iv: "6fc134a5e5851fffa66be6a093ff4f3c"
server_key_version: "encryptionkey@0"
I20260812 02:40:08.833078 28699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:08.851893 28699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:08.855279 28699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:08.855487 28699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:08.856150 28699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:40:08.857077 28699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:40:08.857133 28699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:08.857219 28699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:40:08.857249 28699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:08.866611 28699 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:38943
I20260812 02:40:08.866693 28907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:38943 every 8 connection(s)
I20260812 02:40:08.867733 28699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:40:08.868794 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 28699
I20260812 02:40:08.872409 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:08.870459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.131:33021)
0812 02:40:08.871013 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:08.871139 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":159,"threads_started":1}
W20260812 02:40:08.872900 28908 heartbeater.cc:644] Failed to heartbeat to 127.3.127.190:36149 (0 consecutive failures): Network error: Failed to ping master at 127.3.127.190:36149: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
I20260812 02:40:08.873564 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:08.873344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.131:40089)
0812 02:40:08.873456 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:08.873505 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 02:40:08.874166 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:08.873961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.131:55449)
0812 02:40:08.874069 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:08.874115 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 02:40:08.874337 28908 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 02:40:09.525936 28914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:09.525451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.129:36725)
0812 02:40:09.525749 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:09.525813 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":81,"threads_started":1}
I20260812 02:40:09.673637 28915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:09.673064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.130:50883)
0812 02:40:09.673447 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:09.673512 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":148,"threads_started":1}
I20260812 02:40:09.875485 28916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:09.875003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.131:46167)
0812 02:40:09.875291 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:09.875357 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":81,"threads_started":1}
I20260812 02:40:10.333621 28918 raft_consensus.cc:482] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:40:10.333856 28918 raft_consensus.cc:504] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } } peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } }
I20260812 02:40:10.334983 28918 leader_election.cc:288] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dec61c69893440f7a842a541d868b287 (127.3.127.130:34439), 75b26ee9894d444aa4a42991c9b3f48c (127.3.127.131:38943)
Aug 12 02:40:10 dist-test-slave-rmhg krb5kdc[26167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502408, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.131@KRBTEST.COM
Aug 12 02:40:10 dist-test-slave-rmhg krb5kdc[26167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502408, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.130@KRBTEST.COM
I20260812 02:40:10.353492 28923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:10.335319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:34439 (local address 127.3.127.129:54045)
0812 02:40:10.336015 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:10.336032 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:10.336246 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:10.338338 (+ 2092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:10.338346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:10.338980 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:10.339811 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:10.339838 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:10.342107 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:10.342112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:10.343926 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:10.343947 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:10.345250 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:10.345255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:10.345567 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:10.346419 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:10.346439 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:10.348528 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:10.350696 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:10.350699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:10.350700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:10.351129 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:10.351516 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:10.351519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:10.351520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:10.351688 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:10.352398 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:10.352405 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:10.352672 (+ 267us) client_negotiation.cc:769] Sending connection context
0812 02:40:10.352850 (+ 178us) client_negotiation.cc:240] Negotiation successful
0812 02:40:10.353295 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":362,"spinlock_wait_cycles":58880,"thread_start_us":95,"threads_started":1}
I20260812 02:40:10.353511 28928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:10.335950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:45889 (local address 127.3.127.131:38943)
0812 02:40:10.336396 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:10.336418 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:10.336446 (+ 28us) server_negotiation.cc:407] Connection header received
0812 02:40:10.337363 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:10.337370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:10.337845 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:10.338210 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:10.340017 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:10.341859 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:10.344149 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:10.345033 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:10.345284 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:10.348544 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:10.348551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:10.348568 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:10.348607 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:10.350460 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:10.351261 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:10.351267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:10.351274 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:10.351410 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:10.351803 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:10.351807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:10.351809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:10.352265 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:10.352390 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:10.352994 (+ 604us) server_negotiation.cc:299] Negotiation successful
0812 02:40:10.353304 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":127,"threads_started":1}
I20260812 02:40:10.353514 28924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:10.335501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:54045 (local address 127.3.127.130:34439)
0812 02:40:10.336305 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:10.336322 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:10.336375 (+ 53us) server_negotiation.cc:407] Connection header received
0812 02:40:10.337340 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:10.337351 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:10.337845 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:10.338210 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:10.340017 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:10.341859 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:10.344149 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:10.345033 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:10.345284 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:10.348744 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:10.348750 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:10.348764 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:10.348799 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:10.350578 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:10.351258 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:10.351264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:10.351269 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:10.351404 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:10.351785 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:10.351789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:10.351791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:10.352265 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:10.352389 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:10.352994 (+ 605us) server_negotiation.cc:299] Negotiation successful
0812 02:40:10.353322 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":144,"threads_started":1}
I20260812 02:40:10.353530 28927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:10.335747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:38943 (local address 127.3.127.129:45889)
0812 02:40:10.336102 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:10.336114 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:10.336235 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:10.338338 (+ 2103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:10.338346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:10.338980 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:10.339811 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:10.339838 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:10.342107 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:10.342112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:10.343926 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:10.343947 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:10.345250 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:10.345255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:10.345567 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:10.346393 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:10.346418 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:10.348243 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:10.350661 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:10.350669 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:10.350688 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:10.351122 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:10.351519 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:10.351522 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:10.351524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:10.351698 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:10.352397 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:10.352405 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:10.352671 (+ 266us) client_negotiation.cc:769] Sending connection context
0812 02:40:10.352851 (+ 180us) client_negotiation.cc:240] Negotiation successful
0812 02:40:10.353372 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":444,"spinlock_wait_cycles":30976,"thread_start_us":83,"threads_started":1}
W20260812 02:40:10.356312 28268 leader_election.cc:341] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75b26ee9894d444aa4a42991c9b3f48c (127.3.127.131:38943): Not found: Tablet not found: 37db7236ff3e481090b9d97cf48212e7
W20260812 02:40:10.356892 28268 leader_election.cc:341] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dec61c69893440f7a842a541d868b287 (127.3.127.130:34439): Not found: Tablet not found: 37db7236ff3e481090b9d97cf48212e7
I20260812 02:40:10.356980 28268 leader_election.cc:302] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dc6fd13a07b4963b48c6e0c6324f456; no voters: 75b26ee9894d444aa4a42991c9b3f48c, dec61c69893440f7a842a541d868b287
I20260812 02:40:10.357295 28918 raft_consensus.cc:2726] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 02:40:10.526958 28923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:10.526633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.129:48099)
0812 02:40:10.526800 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:10.526865 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 02:40:10.675015 28929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:10.674467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.130:39521)
0812 02:40:10.674789 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:10.674856 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":84,"threads_started":1}
I20260812 02:40:10.876796 28930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:10.876351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.131:34005)
0812 02:40:10.876617 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:10.876691 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
I20260812 02:40:11.528324 28931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:11.527831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.129:56713)
0812 02:40:11.528150 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:11.528221 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":86,"threads_started":1}
I20260812 02:40:11.676347 28932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:11.675862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.130:38057)
0812 02:40:11.676176 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:11.676244 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":80,"threads_started":1}
I20260812 02:40:11.878187 28934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:11.877680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.131:42815)
0812 02:40:11.877989 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:11.878060 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":92,"threads_started":1}
I20260812 02:40:11.940240 28935 raft_consensus.cc:482] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:40:11.940387 28935 raft_consensus.cc:504] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } } peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } }
I20260812 02:40:11.940878 28935 leader_election.cc:288] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dec61c69893440f7a842a541d868b287 (127.3.127.130:34439), 75b26ee9894d444aa4a42991c9b3f48c (127.3.127.131:38943)
W20260812 02:40:11.941778 28268 leader_election.cc:341] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dec61c69893440f7a842a541d868b287 (127.3.127.130:34439): Not found: Tablet not found: 37db7236ff3e481090b9d97cf48212e7
W20260812 02:40:11.941953 28268 leader_election.cc:341] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75b26ee9894d444aa4a42991c9b3f48c (127.3.127.131:38943): Not found: Tablet not found: 37db7236ff3e481090b9d97cf48212e7
I20260812 02:40:11.942010 28268 leader_election.cc:302] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dc6fd13a07b4963b48c6e0c6324f456; no voters: 75b26ee9894d444aa4a42991c9b3f48c, dec61c69893440f7a842a541d868b287
I20260812 02:40:11.942170 28935 raft_consensus.cc:2726] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 02:40:12.529757 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:12.529253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.129:49311)
0812 02:40:12.529551 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:12.529622 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":82,"threads_started":1}
I20260812 02:40:12.677673 28938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:12.677192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.130:41645)
0812 02:40:12.677501 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:12.677570 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":92,"threads_started":1}
I20260812 02:40:12.879626 28940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:12.879087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.131:43995)
0812 02:40:12.879415 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:12.879486 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":84,"threads_started":1}
I20260812 02:40:13.531430 28941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:13.530864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.129:36975)
0812 02:40:13.531244 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:13.531324 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":82,"threads_started":1}
I20260812 02:40:13.679030 28942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:13.678479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.130:35411)
0812 02:40:13.678835 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:13.678903 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":84,"threads_started":1}
I20260812 02:40:13.880944 28943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:13.880490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.131:59361)
0812 02:40:13.880766 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:13.880839 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
I20260812 02:40:14.532694 28944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:14.532229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.129:46719)
0812 02:40:14.532523 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:14.532587 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":83,"threads_started":1}
I20260812 02:40:14.680378 28945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:14.679867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.130:33427)
0812 02:40:14.680203 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:14.680270 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":100,"threads_started":1}
I20260812 02:40:14.827962 28946 raft_consensus.cc:482] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:40:14.828114 28946 raft_consensus.cc:504] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } } peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } }
I20260812 02:40:14.828567 28946 leader_election.cc:288] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dec61c69893440f7a842a541d868b287 (127.3.127.130:34439), 75b26ee9894d444aa4a42991c9b3f48c (127.3.127.131:38943)
W20260812 02:40:14.829484 28268 leader_election.cc:341] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dec61c69893440f7a842a541d868b287 (127.3.127.130:34439): Not found: Tablet not found: 37db7236ff3e481090b9d97cf48212e7
W20260812 02:40:14.829644 28268 leader_election.cc:341] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75b26ee9894d444aa4a42991c9b3f48c (127.3.127.131:38943): Not found: Tablet not found: 37db7236ff3e481090b9d97cf48212e7
I20260812 02:40:14.829697 28268 leader_election.cc:302] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dc6fd13a07b4963b48c6e0c6324f456; no voters: 75b26ee9894d444aa4a42991c9b3f48c, dec61c69893440f7a842a541d868b287
I20260812 02:40:14.829876 28946 raft_consensus.cc:2726] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 02:40:14.882310 28947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:14.881813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.131:51843)
0812 02:40:14.882118 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:14.882208 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
I20260812 02:40:15.533998 28948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:15.533539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.129:45403)
0812 02:40:15.533829 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:15.533895 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":89,"threads_started":1}
I20260812 02:40:15.681917 28949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:15.681400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.130:35279)
0812 02:40:15.681740 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:15.681809 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":80,"threads_started":1}
I20260812 02:40:15.883759 28951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:15.883231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.131:38711)
0812 02:40:15.883580 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:15.883648 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":95,"threads_started":1}
I20260812 02:40:16.535285 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:16.534834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.129:53469)
0812 02:40:16.535111 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:16.535181 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
I20260812 02:40:16.683383 28953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:16.682848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.130:55243)
0812 02:40:16.683203 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:16.683280 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":84,"threads_started":1}
I20260812 02:40:16.885222 28954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:16.884696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.131:35053)
0812 02:40:16.885003 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:16.885073 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":84,"threads_started":1}
I20260812 02:40:17.536640 28956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:17.536156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.129:40393)
0812 02:40:17.536453 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:17.536528 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":94,"threads_started":1}
I20260812 02:40:17.684917 28957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:17.684377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.130:53929)
0812 02:40:17.684737 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:17.684810 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":123,"threads_started":1}
I20260812 02:40:17.886610 28959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:17.886085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.131:54633)
0812 02:40:17.886401 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:17.886473 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":110,"threads_started":1}
I20260812 02:40:18.414075 28961 raft_consensus.cc:482] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:40:18.414256 28961 raft_consensus.cc:504] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } } peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } }
I20260812 02:40:18.414732 28961 leader_election.cc:288] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dec61c69893440f7a842a541d868b287 (127.3.127.130:34439), 75b26ee9894d444aa4a42991c9b3f48c (127.3.127.131:38943)
W20260812 02:40:18.415546 28268 leader_election.cc:341] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dec61c69893440f7a842a541d868b287 (127.3.127.130:34439): Not found: Tablet not found: 37db7236ff3e481090b9d97cf48212e7
W20260812 02:40:18.415678 28268 leader_election.cc:341] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75b26ee9894d444aa4a42991c9b3f48c (127.3.127.131:38943): Not found: Tablet not found: 37db7236ff3e481090b9d97cf48212e7
I20260812 02:40:18.415722 28268 leader_election.cc:302] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dc6fd13a07b4963b48c6e0c6324f456; no voters: 75b26ee9894d444aa4a42991c9b3f48c, dec61c69893440f7a842a541d868b287
I20260812 02:40:18.415870 28961 raft_consensus.cc:2726] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 02:40:18.538038 28962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:18.537570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.129:60669)
0812 02:40:18.537850 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:18.537913 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":100,"threads_started":1}
I20260812 02:40:18.686506 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:18.685983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.130:60437)
0812 02:40:18.686322 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:18.686398 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":96,"threads_started":1}
I20260812 02:40:18.888024 28967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:18.887532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.131:49747)
0812 02:40:18.887855 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:18.887921 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
I20260812 02:40:19.539274 28968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:19.538803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.129:33539)
0812 02:40:19.539108 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:19.539170 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":78,"threads_started":1}
I20260812 02:40:19.687934 28969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:19.687408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.130:54681)
0812 02:40:19.687753 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:19.687826 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":104,"threads_started":1}
I20260812 02:40:19.889535 28971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:19.888965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.131:38771)
0812 02:40:19.889354 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:19.889429 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":111,"threads_started":1}
I20260812 02:40:20.540797 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:20.540264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.129:45699)
0812 02:40:20.540580 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:20.540687 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":84,"threads_started":1}
I20260812 02:40:20.689296 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:20.688802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.130:32775)
0812 02:40:20.689082 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:20.689141 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
I20260812 02:40:20.890910 28974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:20.890390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.131:41791)
0812 02:40:20.890744 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:20.890807 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":119,"threads_started":1}
I20260812 02:40:21.542184 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:21.541673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.129:42739)
0812 02:40:21.541982 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:21.542050 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":89,"threads_started":1}
I20260812 02:40:21.690725 28978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:21.690161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.130:46441)
0812 02:40:21.690514 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:21.690593 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":84,"threads_started":1}
I20260812 02:40:21.892202 28979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:21.891734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.131:42115)
0812 02:40:21.892034 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:21.892096 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":86,"threads_started":1}
I20260812 02:40:22.386917 28983 raft_consensus.cc:482] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:40:22.387071 28983 raft_consensus.cc:504] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec61c69893440f7a842a541d868b287" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34439 } } peers { permanent_uuid: "75b26ee9894d444aa4a42991c9b3f48c" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 38943 } } peers { permanent_uuid: "9dc6fd13a07b4963b48c6e0c6324f456" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 34143 } }
I20260812 02:40:22.387567 28983 leader_election.cc:288] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dec61c69893440f7a842a541d868b287 (127.3.127.130:34439), 75b26ee9894d444aa4a42991c9b3f48c (127.3.127.131:38943)
W20260812 02:40:22.388474 28268 leader_election.cc:341] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75b26ee9894d444aa4a42991c9b3f48c (127.3.127.131:38943): Not found: Tablet not found: 37db7236ff3e481090b9d97cf48212e7
W20260812 02:40:22.388643 28268 leader_election.cc:341] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dec61c69893440f7a842a541d868b287 (127.3.127.130:34439): Not found: Tablet not found: 37db7236ff3e481090b9d97cf48212e7
I20260812 02:40:22.388693 28268 leader_election.cc:302] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dc6fd13a07b4963b48c6e0c6324f456; no voters: 75b26ee9894d444aa4a42991c9b3f48c, dec61c69893440f7a842a541d868b287
I20260812 02:40:22.388870 28983 raft_consensus.cc:2726] T 37db7236ff3e481090b9d97cf48212e7 P 9dc6fd13a07b4963b48c6e0c6324f456 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 02:40:22.543545 28984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:22.543060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.129:39839)
0812 02:40:22.543369 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:22.543437 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":83,"threads_started":1}
I20260812 02:40:22.692070 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:22.691566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.130:52083)
0812 02:40:22.691887 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:22.691949 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":105,"threads_started":1}
I20260812 02:40:22.893473 28988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:22.892988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.131:38181)
0812 02:40:22.893306 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:22.893370 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
I20260812 02:40:23.545337 28989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:23.544758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.129:35103)
0812 02:40:23.545122 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:23.545234 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":105,"threads_started":1}
I20260812 02:40:23.693679 28990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:23.693147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.130:51447)
0812 02:40:23.693500 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:23.693575 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":82,"threads_started":1}
I20260812 02:40:23.871114 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 28188
I20260812 02:40:23.895076 28995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:23.894598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:36149 (local address 127.3.127.131:60267)
0812 02:40:23.894883 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:23.894951 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
I20260812 02:40:23.896889 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 28461
I20260812 02:40:23.913692 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 28699
2026-08-12T02:40:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19825 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:40:23 dist-test-slave-rmhg krb5kdc[29000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:40:23 dist-test-slave-rmhg krb5kdc[29000](info): set up 2 sockets
Aug 12 02:40:23 dist-test-slave-rmhg krb5kdc[29000](info): commencing operation
krb5kdc: starting...
W20260812 02:40:25.100719 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:40:25 dist-test-slave-rmhg krb5kdc[29000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:40:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:40:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:40:25.240701 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:32981
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:32981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:40:25.363744 29019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:25.364076 29019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:25.364138 29019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:25.368441 29019 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 02:40:25.368521 29019 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 02:40:25.368543 29019 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 02:40:25.368561 29019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:40:25.368585 29019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:25.368610 29019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:25.368633 29019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:40:25.368662 29019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:40:25.374545 29019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:32981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:32981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:25.376085 29019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:25.377606 29019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:25.394248 29037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:25.394371 29038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:25.394577 29019 server_base.cc:1034] running on GCE node
W20260812 02:40:25.394236 29040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:25.395452 29019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:25.397310 29019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:25.398504 29019 hybrid_clock.cc:648] HybridClock initialized: now 1786502425398460 us; error 73 us; skew 500 ppm
Aug 12 02:40:25 dist-test-slave-rmhg krb5kdc[29000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502425, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:25.401924 29019 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:40:25.403204 29019 webserver.cc:460] Webserver started at http://127.3.127.190:40603/ using document root <none> and password file <none>
I20260812 02:40:25.403959 29019 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:25.404062 29019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:25.404317 29019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:40:25.407016 29019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "6b4695b32eda45029e436c362b9ffacd"
format_stamp: "Formatted at 2026-08-12 02:40:25 on dist-test-slave-rmhg"
server_key: "c7ba7395d0e77710c909535a73bb30c8"
server_key_iv: "89031a75abfc5700c3108d2bfdf30177"
server_key_version: "encryptionkey@0"
I20260812 02:40:25.407749 29019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "6b4695b32eda45029e436c362b9ffacd"
format_stamp: "Formatted at 2026-08-12 02:40:25 on dist-test-slave-rmhg"
server_key: "c7ba7395d0e77710c909535a73bb30c8"
server_key_iv: "89031a75abfc5700c3108d2bfdf30177"
server_key_version: "encryptionkey@0"
I20260812 02:40:25.412333 29019 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:40:25.415198 29049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:25.416632 29019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:40:25.416783 29019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "6b4695b32eda45029e436c362b9ffacd"
format_stamp: "Formatted at 2026-08-12 02:40:25 on dist-test-slave-rmhg"
server_key: "c7ba7395d0e77710c909535a73bb30c8"
server_key_iv: "89031a75abfc5700c3108d2bfdf30177"
server_key_version: "encryptionkey@0"
I20260812 02:40:25.416911 29019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:25.431334 29019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:25.435026 29019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:25.435232 29019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:25.442457 29019 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:32981
I20260812 02:40:25.442724 29124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:32981 every 8 connection(s)
I20260812 02:40:25.443506 29019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:40:25.446728 29125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:40:25.449524 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 29019
I20260812 02:40:25.449702 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:40:25.450013 3582 external_mini_cluster.cc:1428] Setting key ed9059bffacd5d3ae323797059911ae2
I20260812 02:40:25.453481 29125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd: Bootstrap starting.
I20260812 02:40:25.456007 29125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd: Neither blocks nor log segments found. Creating new log.
I20260812 02:40:25.456840 29125 log.cc:824] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd: Log is configured to *not* fsync() on all Append() calls
I20260812 02:40:25.459003 29125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd: No bootstrap required, opened a new log
Aug 12 02:40:25 dist-test-slave-rmhg krb5kdc[29000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:40:25.462332 29125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4695b32eda45029e436c362b9ffacd" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 32981 } }
I20260812 02:40:25.462620 29125 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:40:25.462682 29125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4695b32eda45029e436c362b9ffacd, State: Initialized, Role: FOLLOWER
I20260812 02:40:25.463187 29125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4695b32eda45029e436c362b9ffacd" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 32981 } }
I20260812 02:40:25.463347 29125 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:40:25.463445 29125 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:40:25.463558 29125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:40:25.464911 29125 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4695b32eda45029e436c362b9ffacd" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 32981 } }
I20260812 02:40:25.465353 29125 leader_election.cc:302] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b4695b32eda45029e436c362b9ffacd; no voters:
I20260812 02:40:25.465735 29125 leader_election.cc:288] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:40:25.465873 29132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:40:25.466167 29132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 1 LEADER]: Becoming Leader. State: Replica: 6b4695b32eda45029e436c362b9ffacd, State: Running, Role: LEADER
I20260812 02:40:25.466221 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:25.451577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41272 (local address 127.3.127.190:32981)
0812 02:40:25.452173 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:25.452190 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:25.452231 (+ 41us) server_negotiation.cc:407] Connection header received
0812 02:40:25.452919 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:25.452939 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:25.453384 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:25.453738 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:25.454665 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:25.455732 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:25.456707 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:25.457319 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:25.457482 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:25.460064 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:25.460070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:25.460094 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:25.460127 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:25.462343 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:25.462861 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:25.462868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:25.462875 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:25.463009 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:25.463317 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:25.463321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:25.463322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:25.464002 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:25.464235 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:25.464745 (+ 510us) server_negotiation.cc:299] Negotiation successful
0812 02:40:25.465120 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":166,"threads_started":1}
I20260812 02:40:25.466538 29132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4695b32eda45029e436c362b9ffacd" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 32981 } }
I20260812 02:40:25.467175 29125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:40:25.469491 29134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b4695b32eda45029e436c362b9ffacd. Latest consensus state: current_term: 1 leader_uuid: "6b4695b32eda45029e436c362b9ffacd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4695b32eda45029e436c362b9ffacd" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 32981 } } }
I20260812 02:40:25.469648 29134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [sys.catalog]: This master's current role is: LEADER
I20260812 02:40:25.469630 29133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b4695b32eda45029e436c362b9ffacd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4695b32eda45029e436c362b9ffacd" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 32981 } } }
I20260812 02:40:25.469696 29133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [sys.catalog]: This master's current role is: LEADER
W20260812 02:40:25.472507 29156 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:40:25.472602 29156 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:40:25.472702 29157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:40:25.473706 29157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:40:25.480650 29157 catalog_manager.cc:1347] Generated new cluster ID: bc11ec4797f64b9d9ab4e65907b3988d
I20260812 02:40:25.480759 29157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:40:25.494338 29157 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:40:25.495384 29157 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:40:25.502614 29157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd: Generated new TSK 0
I20260812 02:40:25.503479 29157 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
I20260812 02:40:25.535960 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:32981
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:40:25.658634 29171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:25.658936 29171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:25.658999 29171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:25.663134 29171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:25.663209 29171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:25.663293 29171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:40:25.668921 29171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:32981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:25.670173 29171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:25.671289 29171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:25.681299 29182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:25.681331 29188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:25.681293 29181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:25.681457 29171 server_base.cc:1034] running on GCE node
I20260812 02:40:25.682127 29171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:25.682874 29171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:25.684054 29171 hybrid_clock.cc:648] HybridClock initialized: now 1786502425684044 us; error 48 us; skew 500 ppm
Aug 12 02:40:25 dist-test-slave-rmhg krb5kdc[29000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502425, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:25.687137 29171 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:40:25.688174 29171 webserver.cc:460] Webserver started at http://127.3.127.129:45345/ using document root <none> and password file <none>
I20260812 02:40:25.688735 29171 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:25.688796 29171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:25.689008 29171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:40:25.691223 29171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "b21083b8daab4989b997fc4d9d660227"
format_stamp: "Formatted at 2026-08-12 02:40:25 on dist-test-slave-rmhg"
server_key: "5d1556eb453495c340acd458b8d6206f"
server_key_iv: "19e438a457cebe867f6f5fadd528dc74"
server_key_version: "encryptionkey@0"
I20260812 02:40:25.691776 29171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "b21083b8daab4989b997fc4d9d660227"
format_stamp: "Formatted at 2026-08-12 02:40:25 on dist-test-slave-rmhg"
server_key: "5d1556eb453495c340acd458b8d6206f"
server_key_iv: "19e438a457cebe867f6f5fadd528dc74"
server_key_version: "encryptionkey@0"
I20260812 02:40:25.695762 29171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:40:25.698459 29199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:25.699949 29171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:40:25.700098 29171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "b21083b8daab4989b997fc4d9d660227"
format_stamp: "Formatted at 2026-08-12 02:40:25 on dist-test-slave-rmhg"
server_key: "5d1556eb453495c340acd458b8d6206f"
server_key_iv: "19e438a457cebe867f6f5fadd528dc74"
server_key_version: "encryptionkey@0"
I20260812 02:40:25.700237 29171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:25.714504 29171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:25.718024 29171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:25.718214 29171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:25.718859 29171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:40:25.719725 29171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:40:25.719822 29171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:25.719888 29171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:40:25.719918 29171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:25.728993 29171 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:41547
I20260812 02:40:25.729070 29381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:41547 every 8 connection(s)
I20260812 02:40:25.729992 29171 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:40:25.734622 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 29171
I20260812 02:40:25.734752 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:40:25.735045 3582 external_mini_cluster.cc:1428] Setting key 773f7cc16f1ebfe96a86fe7292fc0a45
Aug 12 02:40:25 dist-test-slave-rmhg krb5kdc[29000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502425, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:40:25.744650 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:25.732017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:53253 (local address 127.3.127.190:32981)
0812 02:40:25.732161 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:25.732165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:25.733333 (+ 1168us) server_negotiation.cc:407] Connection header received
0812 02:40:25.734243 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:25.734246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:25.734297 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:25.734388 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:25.735676 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:25.736272 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:25.737644 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:25.737949 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:25.738065 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:25.740839 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:25.740842 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:25.740844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:25.740870 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:25.742199 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:25.742856 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:25.742859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:25.742861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:25.742962 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:25.743393 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:25.743396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:25.743398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:25.743614 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:25.743704 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:25.744364 (+ 660us) server_negotiation.cc:299] Negotiation successful
0812 02:40:25.744492 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 02:40:25.746016 29386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:25.732395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.129:53253)
0812 02:40:25.733175 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:25.733237 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:25.734017 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:25.734550 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:25.734557 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:25.734992 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:25.735549 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:25.735564 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:25.736401 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:25.736405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:25.737451 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:25.737462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:25.738136 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:25.738140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:25.738296 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:25.738901 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:25.738924 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:25.740635 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:25.742342 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:25.742350 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:25.742368 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:25.742732 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:25.743068 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:25.743072 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:25.743074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:25.743271 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:25.743727 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:25.743735 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:25.744114 (+ 379us) client_negotiation.cc:769] Sending connection context
0812 02:40:25.744344 (+ 230us) client_negotiation.cc:240] Negotiation successful
0812 02:40:25.744768 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":131,"threads_started":1}
WARNING: no policy specified for kudu/127.3.127.130@KRBTEST.COM; defaulting to no policy
I20260812 02:40:25.747292 29383 heartbeater.cc:347] Connected to a master server at 127.3.127.190:32981
I20260812 02:40:25.747593 29383 heartbeater.cc:464] Registering TS with master...
I20260812 02:40:25.748176 29383 heartbeater.cc:511] Master 127.3.127.190:32981 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.130@KRBTEST.COM" created.
I20260812 02:40:25.749993 29079 ts_manager.cc:194] Registered new tserver with Master: b21083b8daab4989b997fc4d9d660227 (127.3.127.129:41547)
I20260812 02:40:25.751611 29079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:53253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
I20260812 02:40:25.768738 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:32981
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:40:25.887840 29390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:25.888142 29390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:25.888190 29390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:25.892334 29390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:25.892395 29390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:25.892463 29390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:40:25.897975 29390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:32981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:25.899089 29390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:25.900136 29390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:25.910036 29400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:25.910043 29403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:25.910043 29398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:25.910513 29390 server_base.cc:1034] running on GCE node
I20260812 02:40:25.910930 29390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:25.911623 29390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:25.912781 29390 hybrid_clock.cc:648] HybridClock initialized: now 1786502425912760 us; error 33 us; skew 500 ppm
Aug 12 02:40:25 dist-test-slave-rmhg krb5kdc[29000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502425, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:25.915730 29390 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:40:25.916767 29390 webserver.cc:460] Webserver started at http://127.3.127.130:38711/ using document root <none> and password file <none>
I20260812 02:40:25.917414 29390 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:25.917486 29390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:25.917716 29390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:40:25.919925 29390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "019982c5360e4273a895747e6dec7c03"
format_stamp: "Formatted at 2026-08-12 02:40:25 on dist-test-slave-rmhg"
server_key: "eb31442d86214487313216eeda7ad9e5"
server_key_iv: "f7dd39978cbf602c227f6db29625b3f7"
server_key_version: "encryptionkey@0"
I20260812 02:40:25.920565 29390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "019982c5360e4273a895747e6dec7c03"
format_stamp: "Formatted at 2026-08-12 02:40:25 on dist-test-slave-rmhg"
server_key: "eb31442d86214487313216eeda7ad9e5"
server_key_iv: "f7dd39978cbf602c227f6db29625b3f7"
server_key_version: "encryptionkey@0"
I20260812 02:40:25.924579 29390 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:40:25.927323 29414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:25.928695 29390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:40:25.928812 29390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "019982c5360e4273a895747e6dec7c03"
format_stamp: "Formatted at 2026-08-12 02:40:25 on dist-test-slave-rmhg"
server_key: "eb31442d86214487313216eeda7ad9e5"
server_key_iv: "f7dd39978cbf602c227f6db29625b3f7"
server_key_version: "encryptionkey@0"
I20260812 02:40:25.928938 29390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:25.949435 29390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:25.952865 29390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:25.953078 29390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:25.953751 29390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:40:25.954656 29390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:40:25.954705 29390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:25.954751 29390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:40:25.954782 29390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:25.963889 29390 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:39947
I20260812 02:40:25.963917 29603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:39947 every 8 connection(s)
I20260812 02:40:25.964908 29390 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:40:25.967052 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 29390
I20260812 02:40:25.967175 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:40:25.967487 3582 external_mini_cluster.cc:1428] Setting key c11b6e07ac0b6ead1b183cc4f050f3cf
Aug 12 02:40:25 dist-test-slave-rmhg krb5kdc[29000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502425, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.127.131@KRBTEST.COM; defaulting to no policy
I20260812 02:40:25.980772 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:25.966935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:48719 (local address 127.3.127.190:32981)
0812 02:40:25.967069 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:25.967072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:25.968656 (+ 1584us) server_negotiation.cc:407] Connection header received
0812 02:40:25.969766 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:25.969770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:25.969845 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:25.969934 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:25.971417 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:25.971959 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:25.973413 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:25.973736 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:25.973845 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:25.976717 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:25.976720 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:25.976723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:25.976753 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:25.978262 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:25.978951 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:25.978954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:25.978956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:25.979066 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:25.979454 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:25.979458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:25.979460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:25.979740 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:25.979835 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:25.980411 (+ 576us) server_negotiation.cc:299] Negotiation successful
0812 02:40:25.980636 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.131@KRBTEST.COM" created.
I20260812 02:40:25.981864 29607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:25.967512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.130:48719)
0812 02:40:25.968493 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:25.968553 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:25.969517 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:25.970230 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:25.970238 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:25.970629 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:25.971250 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:25.971272 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:25.972095 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:25.972099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:25.973285 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:25.973297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:25.973894 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:25.973897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:25.974091 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:25.974749 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:25.974768 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:25.976474 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:25.978408 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:25.978415 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:25.978436 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:25.978833 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:25.979148 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:25.979151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:25.979152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:25.979325 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:25.979838 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:25.979844 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:25.980142 (+ 298us) client_negotiation.cc:769] Sending connection context
0812 02:40:25.980315 (+ 173us) client_negotiation.cc:240] Negotiation successful
0812 02:40:25.980654 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":169,"threads_started":1}
I20260812 02:40:25.983148 29604 heartbeater.cc:347] Connected to a master server at 127.3.127.190:32981
I20260812 02:40:25.983400 29604 heartbeater.cc:464] Registering TS with master...
I20260812 02:40:25.984086 29604 heartbeater.cc:511] Master 127.3.127.190:32981 requested a full tablet report, sending...
I20260812 02:40:25.985411 29079 ts_manager.cc:194] Registered new tserver with Master: 019982c5360e4273a895747e6dec7c03 (127.3.127.130:39947)
I20260812 02:40:25.986034 29079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:48719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
I20260812 02:40:25.998338 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:32981
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:40:26.117728 29610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:26.118044 29610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:26.118108 29610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:26.122395 29610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:26.122483 29610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:26.122568 29610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:40:26.128250 29610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:32981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:26.129573 29610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:26.130674 29610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:26.140967 29616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:26.140995 29619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:26.140967 29617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:26.141268 29610 server_base.cc:1034] running on GCE node
I20260812 02:40:26.141734 29610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:26.142427 29610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:26.143658 29610 hybrid_clock.cc:648] HybridClock initialized: now 1786502426143638 us; error 39 us; skew 500 ppm
Aug 12 02:40:26 dist-test-slave-rmhg krb5kdc[29000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502426, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:26.146668 29610 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:40:26.147650 29610 webserver.cc:460] Webserver started at http://127.3.127.131:38583/ using document root <none> and password file <none>
I20260812 02:40:26.148252 29610 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:26.148329 29610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:26.148550 29610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:40:26.150789 29610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "af2edfdfdc664199b20e4efdc6656c48"
format_stamp: "Formatted at 2026-08-12 02:40:26 on dist-test-slave-rmhg"
server_key: "ce4a7a8d24f9b731f76976d701f547d1"
server_key_iv: "97b101e642448cc497cd9c959a70a50b"
server_key_version: "encryptionkey@0"
I20260812 02:40:26.151440 29610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "af2edfdfdc664199b20e4efdc6656c48"
format_stamp: "Formatted at 2026-08-12 02:40:26 on dist-test-slave-rmhg"
server_key: "ce4a7a8d24f9b731f76976d701f547d1"
server_key_iv: "97b101e642448cc497cd9c959a70a50b"
server_key_version: "encryptionkey@0"
I20260812 02:40:26.155527 29610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:40:26.158334 29631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:26.159770 29610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:40:26.159905 29610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "af2edfdfdc664199b20e4efdc6656c48"
format_stamp: "Formatted at 2026-08-12 02:40:26 on dist-test-slave-rmhg"
server_key: "ce4a7a8d24f9b731f76976d701f547d1"
server_key_iv: "97b101e642448cc497cd9c959a70a50b"
server_key_version: "encryptionkey@0"
I20260812 02:40:26.160028 29610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:26.172348 29610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:26.175643 29610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:26.175841 29610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:26.176481 29610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:40:26.177412 29610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:40:26.177475 29610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:26.177533 29610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:40:26.177559 29610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:26.187281 29610 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:37377
I20260812 02:40:26.187330 29819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:37377 every 8 connection(s)
I20260812 02:40:26.188292 29610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:40:26.197234 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 29610
I20260812 02:40:26.197386 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:40:26.197772 3582 external_mini_cluster.cc:1428] Setting key e46050a70ed39d1bdd435cfd2bdf6dfb
Aug 12 02:40:26 dist-test-slave-rmhg krb5kdc[29000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502426, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:40:26.203210 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:26.190398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:40807 (local address 127.3.127.190:32981)
0812 02:40:26.190541 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:26.190546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:26.191804 (+ 1258us) server_negotiation.cc:407] Connection header received
0812 02:40:26.192762 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:26.192765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:26.192829 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:26.192905 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:26.194270 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.194817 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.196008 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.196242 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.196339 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:26.199442 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:26.199447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:26.199450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:26.199480 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:26.200916 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:26.201543 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:26.201548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:26.201550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:26.201639 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:26.201999 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:26.202002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:26.202005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:26.202218 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:26.202312 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:26.202884 (+ 572us) server_negotiation.cc:299] Negotiation successful
0812 02:40:26.203044 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 02:40:26.204084 29824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:26.190864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.131:40807)
0812 02:40:26.191628 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:26.191679 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:26.192529 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:26.193114 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:26.193122 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:26.193576 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:26.194112 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:26.194128 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.194947 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.194949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:26.195878 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:26.195889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.196406 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.196410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:26.196596 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:26.197298 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:26.197321 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:26.199258 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:26.201051 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:26.201058 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:26.201078 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:26.201438 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:26.201728 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:26.201732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:26.201734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:26.201893 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:26.202327 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:26.202334 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:26.202628 (+ 294us) client_negotiation.cc:769] Sending connection context
0812 02:40:26.202804 (+ 176us) client_negotiation.cc:240] Negotiation successful
0812 02:40:26.203199 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":138,"threads_started":1}
I20260812 02:40:26.205330 29822 heartbeater.cc:347] Connected to a master server at 127.3.127.190:32981
I20260812 02:40:26.205622 29822 heartbeater.cc:464] Registering TS with master...
I20260812 02:40:26.206161 29822 heartbeater.cc:511] Master 127.3.127.190:32981 requested a full tablet report, sending...
I20260812 02:40:26.207310 29079 ts_manager.cc:194] Registered new tserver with Master: af2edfdfdc664199b20e4efdc6656c48 (127.3.127.131:37377)
I20260812 02:40:26.207932 29079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:40807
I20260812 02:40:26.212283 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:40:26.222610 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:26.214439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41274 (local address 127.3.127.190:32981)
0812 02:40:26.214594 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:26.214598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:26.214762 (+ 164us) server_negotiation.cc:407] Connection header received
0812 02:40:26.214862 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:26.214864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:26.214916 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:26.214992 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:26.215717 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.216265 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.217202 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.217426 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.217508 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:26.219031 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:26.219034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:26.219037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:26.219066 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:26.220501 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:26.221142 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:26.221146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:26.221195 (+ 49us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:26.221322 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:26.221595 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:26.221597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:26.221598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:26.221899 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:26.222038 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:26.222314 (+ 276us) server_negotiation.cc:299] Negotiation successful
0812 02:40:26.222435 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 02:40:26.226265 29079 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:40:26.229498 29079 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:40:26.246726 29839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:26.240535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45106 (local address 127.3.127.130:39947)
0812 02:40:26.241430 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:26.241442 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:26.241468 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:40:26.241553 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:26.241559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:26.241799 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:26.241974 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:40:26.242549 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.243314 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.245271 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.245897 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.246017 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:26.246226 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:26.246414 (+ 188us) server_negotiation.cc:299] Negotiation successful
0812 02:40:26.246533 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":177,"threads_started":1}
I20260812 02:40:26.246722 29838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:26.239834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:41547 (local address 127.0.0.1:41132)
0812 02:40:26.240857 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:26.240924 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:26.241085 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:26.241972 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:26.241978 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:26.242001 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:40:26.242178 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:26.242187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.243775 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.243779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:26.244934 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:26.244947 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.246165 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.246169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:26.246272 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:26.246296 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 02:40:26.246351 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 02:40:26.246535 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":69,"thread_start_us":91,"threads_started":1}
I20260812 02:40:26.246718 29837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:26.239997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:39947 (local address 127.0.0.1:45106)
0812 02:40:26.241038 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:26.241051 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:26.241127 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:26.242178 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:26.242181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:26.242213 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:40:26.242404 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:26.242410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.243452 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.243455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:26.245096 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:26.245110 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.246048 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.246052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:26.246197 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:26.246244 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 02:40:26.246364 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 02:40:26.246537 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":212,"spinlock_wait_cycles":7168,"thread_start_us":123,"threads_started":1}
I20260812 02:40:26.247249 29842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:26.240817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39422 (local address 127.3.127.131:37377)
0812 02:40:26.241311 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:26.241325 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:26.241649 (+ 324us) server_negotiation.cc:407] Connection header received
0812 02:40:26.241772 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:26.241777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:26.241956 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:26.242114 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:40:26.242528 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.243693 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.245240 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.246037 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.246137 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:26.246301 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:26.246987 (+ 686us) server_negotiation.cc:299] Negotiation successful
0812 02:40:26.247107 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":151,"threads_started":1}
I20260812 02:40:26.247406 29835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:26.240271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:37377 (local address 127.0.0.1:39422)
0812 02:40:26.241543 (+ 1272us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:26.241557 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:26.241662 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:26.242178 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:26.242181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:26.242215 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:40:26.242399 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:26.242406 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.243834 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.243836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:26.245116 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:26.245123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.246365 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.246369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:26.246447 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:26.246468 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 02:40:26.246532 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 02:40:26.246605 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1210,"mutex_wait_us":104,"spinlock_wait_cycles":3968,"thread_start_us":124,"threads_started":1}
I20260812 02:40:26.247846 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:26.239997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41132 (local address 127.3.127.129:41547)
0812 02:40:26.241441 (+ 1444us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:26.241456 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:26.241475 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:40:26.241553 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:26.241559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:26.241799 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:26.241974 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:40:26.242329 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.243600 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.245080 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.245991 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.246801 (+ 810us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:26.246974 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:26.247022 (+ 48us) server_negotiation.cc:299] Negotiation successful
0812 02:40:26.247722 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1324,"thread_start_us":109,"threads_started":1}
I20260812 02:40:26.250638 29714 tablet_service.cc:1467] Processing CreateTablet for tablet 5476e811dff84fd09a14e1f96199fa1d (DEFAULT_TABLE table=test-table [id=67025033a82142c5b32987dc598cb950]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:40:26.250670 29279 tablet_service.cc:1467] Processing CreateTablet for tablet 5476e811dff84fd09a14e1f96199fa1d (DEFAULT_TABLE table=test-table [id=67025033a82142c5b32987dc598cb950]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:40:26.250644 29485 tablet_service.cc:1467] Processing CreateTablet for tablet 5476e811dff84fd09a14e1f96199fa1d (DEFAULT_TABLE table=test-table [id=67025033a82142c5b32987dc598cb950]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:40:26.251796 29279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5476e811dff84fd09a14e1f96199fa1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:40:26.251807 29714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5476e811dff84fd09a14e1f96199fa1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:40:26.252167 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5476e811dff84fd09a14e1f96199fa1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:40:26.258517 29846 tablet_bootstrap.cc:492] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48: Bootstrap starting.
I20260812 02:40:26.260056 29847 tablet_bootstrap.cc:492] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227: Bootstrap starting.
I20260812 02:40:26.261483 29848 tablet_bootstrap.cc:492] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03: Bootstrap starting.
I20260812 02:40:26.261631 29846 tablet_bootstrap.cc:654] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48: Neither blocks nor log segments found. Creating new log.
I20260812 02:40:26.262465 29847 tablet_bootstrap.cc:654] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227: Neither blocks nor log segments found. Creating new log.
I20260812 02:40:26.262594 29846 log.cc:824] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48: Log is configured to *not* fsync() on all Append() calls
I20260812 02:40:26.263371 29847 log.cc:824] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227: Log is configured to *not* fsync() on all Append() calls
I20260812 02:40:26.263926 29848 tablet_bootstrap.cc:654] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03: Neither blocks nor log segments found. Creating new log.
I20260812 02:40:26.264748 29848 log.cc:824] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03: Log is configured to *not* fsync() on all Append() calls
I20260812 02:40:26.264844 29846 tablet_bootstrap.cc:492] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48: No bootstrap required, opened a new log
I20260812 02:40:26.265054 29846 ts_tablet_manager.cc:1397] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260812 02:40:26.265986 29847 tablet_bootstrap.cc:492] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227: No bootstrap required, opened a new log
I20260812 02:40:26.266201 29847 ts_tablet_manager.cc:1397] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 02:40:26.266533 29848 tablet_bootstrap.cc:492] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03: No bootstrap required, opened a new log
I20260812 02:40:26.266706 29848 ts_tablet_manager.cc:1397] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 02:40:26.269253 29846 raft_consensus.cc:348] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } }
I20260812 02:40:26.269599 29846 raft_consensus.cc:374] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:40:26.269655 29846 raft_consensus.cc:729] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af2edfdfdc664199b20e4efdc6656c48, State: Initialized, Role: FOLLOWER
I20260812 02:40:26.270211 29846 consensus_queue.cc:260] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } }
I20260812 02:40:26.270102 29848 raft_consensus.cc:348] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } }
I20260812 02:40:26.270402 29848 raft_consensus.cc:374] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:40:26.270465 29848 raft_consensus.cc:729] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 019982c5360e4273a895747e6dec7c03, State: Initialized, Role: FOLLOWER
I20260812 02:40:26.270956 29848 consensus_queue.cc:260] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } }
I20260812 02:40:26.270925 29847 raft_consensus.cc:348] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } }
I20260812 02:40:26.271174 29847 raft_consensus.cc:374] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:40:26.271241 29847 raft_consensus.cc:729] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b21083b8daab4989b997fc4d9d660227, State: Initialized, Role: FOLLOWER
I20260812 02:40:26.271435 29846 ts_tablet_manager.cc:1428] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 02:40:26.271679 29847 consensus_queue.cc:260] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } }
I20260812 02:40:26.271864 29822 heartbeater.cc:503] Master 127.3.127.190:32981 was elected leader, sending a full tablet report...
I20260812 02:40:26.271935 29848 ts_tablet_manager.cc:1428] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 02:40:26.272181 29604 heartbeater.cc:503] Master 127.3.127.190:32981 was elected leader, sending a full tablet report...
I20260812 02:40:26.272496 29383 heartbeater.cc:503] Master 127.3.127.190:32981 was elected leader, sending a full tablet report...
I20260812 02:40:26.272648 29847 ts_tablet_manager.cc:1428] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260812 02:40:26.439363 29823 tablet.cc:2369] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:40:26.466296 29606 tablet.cc:2369] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:40:26.481423 29385 tablet.cc:2369] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:40:26.565009 29855 raft_consensus.cc:482] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:40:26.565297 29855 raft_consensus.cc:504] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } }
I20260812 02:40:26.566563 29855 leader_election.cc:288] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b21083b8daab4989b997fc4d9d660227 (127.3.127.129:41547), af2edfdfdc664199b20e4efdc6656c48 (127.3.127.131:37377)
I20260812 02:40:26.570894 29842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:26.566982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:60333 (local address 127.3.127.131:37377)
0812 02:40:26.567122 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:26.567126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:26.567382 (+ 256us) server_negotiation.cc:407] Connection header received
0812 02:40:26.567514 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:26.567517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:26.567568 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:26.567636 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:40:26.568032 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.568646 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.569913 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.570329 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.570446 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:26.570510 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:26.570677 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 02:40:26.570757 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 02:40:26.570904 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:26.567043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:47939 (local address 127.3.127.129:41547)
0812 02:40:26.567160 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:26.567165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:26.567313 (+ 148us) server_negotiation.cc:407] Connection header received
0812 02:40:26.567443 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:26.567446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:26.567512 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:26.567570 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:40:26.567868 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.568439 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.569901 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.570336 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.570454 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:26.570521 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:26.570709 (+ 188us) server_negotiation.cc:299] Negotiation successful
0812 02:40:26.570787 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 02:40:26.570907 29857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:26.566856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:41547 (local address 127.3.127.130:47939)
0812 02:40:26.567220 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:26.567235 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:26.567341 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:26.567624 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:26.567627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:26.567644 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:40:26.567772 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:26.567777 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.568554 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.568557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:26.569760 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:26.569772 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.570460 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.570464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:26.570579 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:26.570598 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 02:40:26.570681 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 02:40:26.570752 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":78,"threads_started":1}
I20260812 02:40:26.570926 29858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:26.566860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:37377 (local address 127.3.127.130:60333)
0812 02:40:26.567263 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:26.567275 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:26.567403 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:26.567685 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:26.567688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:26.567706 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:40:26.567880 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:26.567889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.568780 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.568784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:26.569772 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:26.569783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.570454 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.570458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:26.570553 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:26.570579 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 02:40:26.570643 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 02:40:26.570720 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":96,"thread_start_us":118,"threads_started":1}
I20260812 02:40:26.571553 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5476e811dff84fd09a14e1f96199fa1d" candidate_uuid: "019982c5360e4273a895747e6dec7c03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b21083b8daab4989b997fc4d9d660227" is_pre_election: true
I20260812 02:40:26.571691 29748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5476e811dff84fd09a14e1f96199fa1d" candidate_uuid: "019982c5360e4273a895747e6dec7c03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af2edfdfdc664199b20e4efdc6656c48" is_pre_election: true
I20260812 02:40:26.571888 29307 raft_consensus.cc:2454] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 019982c5360e4273a895747e6dec7c03 in term 0.
I20260812 02:40:26.572011 29748 raft_consensus.cc:2454] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 019982c5360e4273a895747e6dec7c03 in term 0.
I20260812 02:40:26.572405 29417 leader_election.cc:302] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019982c5360e4273a895747e6dec7c03, b21083b8daab4989b997fc4d9d660227; no voters:
I20260812 02:40:26.572698 29855 raft_consensus.cc:2781] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:40:26.572788 29855 raft_consensus.cc:482] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:40:26.572831 29855 raft_consensus.cc:3037] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:40:26.573933 29855 raft_consensus.cc:504] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } }
I20260812 02:40:26.574303 29855 leader_election.cc:288] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 [CANDIDATE]: Term 1 election: Requested vote from peers b21083b8daab4989b997fc4d9d660227 (127.3.127.129:41547), af2edfdfdc664199b20e4efdc6656c48 (127.3.127.131:37377)
I20260812 02:40:26.574723 29748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5476e811dff84fd09a14e1f96199fa1d" candidate_uuid: "019982c5360e4273a895747e6dec7c03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af2edfdfdc664199b20e4efdc6656c48"
I20260812 02:40:26.574764 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5476e811dff84fd09a14e1f96199fa1d" candidate_uuid: "019982c5360e4273a895747e6dec7c03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b21083b8daab4989b997fc4d9d660227"
I20260812 02:40:26.574862 29748 raft_consensus.cc:3037] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:40:26.574885 29307 raft_consensus.cc:3037] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:40:26.575971 29307 raft_consensus.cc:2454] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 019982c5360e4273a895747e6dec7c03 in term 1.
I20260812 02:40:26.575980 29748 raft_consensus.cc:2454] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 019982c5360e4273a895747e6dec7c03 in term 1.
I20260812 02:40:26.576308 29419 leader_election.cc:302] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 019982c5360e4273a895747e6dec7c03, af2edfdfdc664199b20e4efdc6656c48, b21083b8daab4989b997fc4d9d660227; no voters:
I20260812 02:40:26.576505 29855 raft_consensus.cc:2781] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:40:26.576756 29855 raft_consensus.cc:686] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 [term 1 LEADER]: Becoming Leader. State: Replica: 019982c5360e4273a895747e6dec7c03, State: Running, Role: LEADER
I20260812 02:40:26.577129 29855 consensus_queue.cc:237] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } }
I20260812 02:40:26.580426 29079 catalog_manager.cc:5495] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 019982c5360e4273a895747e6dec7c03 (127.3.127.130). New cstate: current_term: 1 leader_uuid: "019982c5360e4273a895747e6dec7c03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } health_report { overall_health: UNKNOWN } } }
I20260812 02:40:26.613178 29839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:26.609443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.3.127.130:39947)
0812 02:40:26.609596 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:26.609601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:26.609617 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:40:26.609742 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:26.609745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:26.609810 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:26.609905 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:40:26.610186 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.610737 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.611607 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:26.611835 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:26.611921 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:26.612249 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:40:26.612735 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:40:26.612846 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:26.612939 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 02:40:26.613002 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 02:40:26.619237 29307 raft_consensus.cc:1261] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Refusing update from remote peer 019982c5360e4273a895747e6dec7c03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:40:26.619427 29748 raft_consensus.cc:1261] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48 [term 1 FOLLOWER]: Refusing update from remote peer 019982c5360e4273a895747e6dec7c03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:40:26.619959 29855 consensus_queue.cc:1035] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:40:26.620150 29859 consensus_queue.cc:1035] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:40:26.629938 29863 mvcc.cc:205] Tried to move back new op lower bound from 7317513939425177600 to 7317513939262783488. Current Snapshot: MvccSnapshot[applied={T|T < 7317513939425177600}]
I20260812 02:40:26.638669 29078 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:40:26.638844 29078 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:40:26.641669 29078 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd: Sending DeleteTablet for 3 replicas of tablet 5476e811dff84fd09a14e1f96199fa1d
I20260812 02:40:26.642590 29279 tablet_service.cc:1514] Processing DeleteTablet for tablet 5476e811dff84fd09a14e1f96199fa1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:40:26 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:41132
I20260812 02:40:26.642580 29485 tablet_service.cc:1514] Processing DeleteTablet for tablet 5476e811dff84fd09a14e1f96199fa1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:40:26 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:45106
I20260812 02:40:26.642966 29714 tablet_service.cc:1514] Processing DeleteTablet for tablet 5476e811dff84fd09a14e1f96199fa1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:40:26 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:39422
I20260812 02:40:26.643452 29878 tablet_replica.cc:330] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48: stopping tablet replica
I20260812 02:40:26.643461 29876 tablet_replica.cc:330] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03: stopping tablet replica
I20260812 02:40:26.643512 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 29171
I20260812 02:40:26.643860 29876 raft_consensus.cc:2229] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:40:26.643862 29878 raft_consensus.cc:2229] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:40:26.644302 29876 raft_consensus.cc:2258] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:40:26.644403 29878 raft_consensus.cc:2258] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:40:26.656002 29876 ts_tablet_manager.cc:1905] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:40:26.656169 29878 ts_tablet_manager.cc:1905] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:40:26.659057 29876 ts_tablet_manager.cc:1918] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:40:26.659205 29876 log.cc:1196] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/5476e811dff84fd09a14e1f96199fa1d
I20260812 02:40:26.659197 29878 ts_tablet_manager.cc:1918] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:40:26.659320 29878 log.cc:1196] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/5476e811dff84fd09a14e1f96199fa1d
I20260812 02:40:26.659574 29876 ts_tablet_manager.cc:1939] T 5476e811dff84fd09a14e1f96199fa1d P 019982c5360e4273a895747e6dec7c03: Deleting consensus metadata
I20260812 02:40:26.659611 29878 ts_tablet_manager.cc:1939] T 5476e811dff84fd09a14e1f96199fa1d P af2edfdfdc664199b20e4efdc6656c48: Deleting consensus metadata
I20260812 02:40:26.660964 29050 catalog_manager.cc:4841] TS 019982c5360e4273a895747e6dec7c03 (127.3.127.130:39947): tablet 5476e811dff84fd09a14e1f96199fa1d (table test-table [id=67025033a82142c5b32987dc598cb950]) successfully deleted
I20260812 02:40:26.660961 29055 catalog_manager.cc:4841] TS af2edfdfdc664199b20e4efdc6656c48 (127.3.127.131:37377): tablet 5476e811dff84fd09a14e1f96199fa1d (table test-table [id=67025033a82142c5b32987dc598cb950]) successfully deleted
W20260812 02:40:26.662729 29053 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.127.129:41547 (error 108)
I20260812 02:40:26.663619 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 29390
I20260812 02:40:26.663836 29835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:26.663532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:41547 (local address 127.0.0.1:41138)
0812 02:40:26.663665 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:26.663748 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.129:41547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 02:40:26.681476 29053 catalog_manager.cc:4584] TS b21083b8daab4989b997fc4d9d660227 (127.3.127.129:41547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5476e811dff84fd09a14e1f96199fa1d: Network error: Client connection negotiation failed: client connection to 127.3.127.129:41547: connect: Connection refused (error 111)
I20260812 02:40:26.687368 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 29610
I20260812 02:40:26.706321 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 29019
I20260812 02:40:46.722081 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:32981
--webserver_interface=127.3.127.190
--webserver_port=40603
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:32981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:40:46.844620 29881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:46.844935 29881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:46.844983 29881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:46.849217 29881 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 02:40:46.849294 29881 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 02:40:46.849310 29881 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 02:40:46.849323 29881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:40:46.849341 29881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:46.849359 29881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:46.849380 29881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:40:46.849408 29881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:40:46.855098 29881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:32981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:32981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=40603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:46.856320 29881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:46.857499 29881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:46.872747 29891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:46.872875 29890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:46.872792 29898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:46.873093 29881 server_base.cc:1034] running on GCE node
I20260812 02:40:46.873843 29881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:46.875293 29881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:46.876536 29881 hybrid_clock.cc:648] HybridClock initialized: now 1786502446876522 us; error 72 us; skew 500 ppm
Aug 12 02:40:46 dist-test-slave-rmhg krb5kdc[29000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:46.887086 29881 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:40:46.888263 29881 webserver.cc:460] Webserver started at http://127.3.127.190:40603/ using document root <none> and password file <none>
I20260812 02:40:46.888912 29881 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:46.889007 29881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:46.893479 29881 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:40:46.895980 29908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:46.897464 29881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:40:46.897619 29881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "6b4695b32eda45029e436c362b9ffacd"
format_stamp: "Formatted at 2026-08-12 02:40:25 on dist-test-slave-rmhg"
server_key: "c7ba7395d0e77710c909535a73bb30c8"
server_key_iv: "89031a75abfc5700c3108d2bfdf30177"
server_key_version: "encryptionkey@0"
I20260812 02:40:46.898285 29881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:46.913940 29881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:46.917651 29881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:46.917877 29881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:46.925931 29881 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:32981
I20260812 02:40:46.926203 29999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:32981 every 8 connection(s)
I20260812 02:40:46.927189 29881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:40:46.931416 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 29881
I20260812 02:40:46.931962 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:41547
--local_ip_for_outbound_sockets=127.3.127.129
--tserver_master_addrs=127.3.127.190:32981
--webserver_port=45345
--webserver_interface=127.3.127.129
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:40:46.932611 30002 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:40:46.934895 30002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd: Bootstrap starting.
I20260812 02:40:46.954044 30002 log.cc:824] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd: Log is configured to *not* fsync() on all Append() calls
I20260812 02:40:46.964003 30002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:40:46.964617 30002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd: Bootstrap complete.
I20260812 02:40:46.970700 30002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4695b32eda45029e436c362b9ffacd" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 32981 } }
I20260812 02:40:46.971207 30002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4695b32eda45029e436c362b9ffacd, State: Initialized, Role: FOLLOWER
I20260812 02:40:46.971819 30002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4695b32eda45029e436c362b9ffacd" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 32981 } }
I20260812 02:40:46.971940 30002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:40:46.971992 30002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:40:46.972066 30002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 1 FOLLOWER]: Advancing to term 2
I20260812 02:40:46.975386 30002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4695b32eda45029e436c362b9ffacd" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 32981 } }
I20260812 02:40:46.975924 30002 leader_election.cc:302] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b4695b32eda45029e436c362b9ffacd; no voters:
I20260812 02:40:46.976944 30002 leader_election.cc:288] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 02:40:46.977200 30013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 2 FOLLOWER]: Leader election won for term 2
I20260812 02:40:46.977535 30013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 2 LEADER]: Becoming Leader. State: Replica: 6b4695b32eda45029e436c362b9ffacd, State: Running, Role: LEADER
I20260812 02:40:46.977954 30013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4695b32eda45029e436c362b9ffacd" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 32981 } }
I20260812 02:40:46.978470 30002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:40:46.979207 30013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b4695b32eda45029e436c362b9ffacd. Latest consensus state: current_term: 2 leader_uuid: "6b4695b32eda45029e436c362b9ffacd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4695b32eda45029e436c362b9ffacd" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 32981 } } }
I20260812 02:40:46.979455 30013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [sys.catalog]: This master's current role is: LEADER
I20260812 02:40:46.979629 30015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6b4695b32eda45029e436c362b9ffacd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4695b32eda45029e436c362b9ffacd" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 32981 } } }
I20260812 02:40:46.979753 30015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [sys.catalog]: This master's current role is: LEADER
I20260812 02:40:46.981333 30024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:40:46.986393 30024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:40:46.987093 30024 catalog_manager.cc:1259] Loaded cluster ID: bc11ec4797f64b9d9ab4e65907b3988d
I20260812 02:40:46.987175 30024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:40:46.989897 30024 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:40:46.990777 30024 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd: Loaded TSK: 0
I20260812 02:40:47.007143 30024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd: Generated new TSK 1
I20260812 02:40:47.007519 30024 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 02:40:47.063098 30005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:47.063406 30005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:47.063473 30005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:47.067693 30005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:47.067787 30005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:47.067874 30005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:40:47.073686 30005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:41547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=45345
--tserver_master_addrs=127.3.127.190:32981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:47.074837 30005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:47.075923 30005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:47.086347 30044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:47.086369 30045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:47.086400 30051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:47.087203 30005 server_base.cc:1034] running on GCE node
I20260812 02:40:47.087647 30005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:47.088353 30005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:47.089524 30005 hybrid_clock.cc:648] HybridClock initialized: now 1786502447089499 us; error 41 us; skew 500 ppm
Aug 12 02:40:47 dist-test-slave-rmhg krb5kdc[29000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502447, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:47.092417 30005 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:40:47.093456 30005 webserver.cc:460] Webserver started at http://127.3.127.129:45345/ using document root <none> and password file <none>
I20260812 02:40:47.094026 30005 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:47.094117 30005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:47.098429 30005 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:40:47.101019 30062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:47.102766 30005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:40:47.102908 30005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "b21083b8daab4989b997fc4d9d660227"
format_stamp: "Formatted at 2026-08-12 02:40:25 on dist-test-slave-rmhg"
server_key: "5d1556eb453495c340acd458b8d6206f"
server_key_iv: "19e438a457cebe867f6f5fadd528dc74"
server_key_version: "encryptionkey@0"
I20260812 02:40:47.103536 30005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:47.116707 30005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:47.120437 30005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:47.120653 30005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:47.121351 30005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:40:47.122555 30081 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 02:40:47.126876 30005 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 02:40:47.126938 30005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260812 02:40:47.126992 30005 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 02:40:47.128762 30005 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 02:40:47.128827 30005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260812 02:40:47.129204 30081 tablet_bootstrap.cc:492] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227: Bootstrap starting.
I20260812 02:40:47.140254 30005 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:41547
I20260812 02:40:47.140337 30254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:41547 every 8 connection(s)
I20260812 02:40:47.141673 30005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:40:47.150769 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 30005
I20260812 02:40:47.151324 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:39947
--local_ip_for_outbound_sockets=127.3.127.130
--tserver_master_addrs=127.3.127.190:32981
--webserver_port=38711
--webserver_interface=127.3.127.130
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:40:47.151505 30081 log.cc:824] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:40:47 dist-test-slave-rmhg krb5kdc[29000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502447, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:40:47.160082 30260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.144123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.129:33909)
0812 02:40:47.144720 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:47.144774 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:47.145680 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:47.146875 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:47.146884 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:47.147337 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:47.147841 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:47.147857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.149272 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.149275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:47.150189 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:47.150199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.151164 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.151169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:47.151385 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.152006 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:47.152026 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:47.153558 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:47.156255 (+ 2697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:47.156262 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:47.156278 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:47.156686 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:47.157109 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:47.157112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:47.157114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:47.157335 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:47.158080 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:47.158087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:47.158425 (+ 338us) client_negotiation.cc:769] Sending connection context
0812 02:40:47.158644 (+ 219us) client_negotiation.cc:240] Negotiation successful
0812 02:40:47.159044 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":151,"threads_started":1}
I20260812 02:40:47.160012 30258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.144054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:33909 (local address 127.3.127.190:32981)
0812 02:40:47.144438 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:47.144478 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:47.144897 (+ 419us) server_negotiation.cc:407] Connection header received
0812 02:40:47.146133 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:47.146145 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:47.146455 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:47.146789 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:47.148024 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.149053 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.150320 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.151004 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.151183 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.153853 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:47.153860 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:47.153879 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:47.153915 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:47.156115 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:47.156829 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:47.156835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:47.156840 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:47.157002 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:47.157447 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:47.157451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:47.157453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:47.157950 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:47.158099 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:47.158808 (+ 709us) server_negotiation.cc:299] Negotiation successful
0812 02:40:47.159132 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":113,"threads_started":1}
I20260812 02:40:47.160665 30081 tablet_bootstrap.cc:492] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:40:47.161077 30081 tablet_bootstrap.cc:492] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227: Bootstrap complete.
I20260812 02:40:47.162431 30081 ts_tablet_manager.cc:1397] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227: Time spent bootstrapping tablet: real 0.034s user 0.021s sys 0.011s
I20260812 02:40:47.163432 30255 heartbeater.cc:347] Connected to a master server at 127.3.127.190:32981
I20260812 02:40:47.163683 30255 heartbeater.cc:464] Registering TS with master...
I20260812 02:40:47.164292 30255 heartbeater.cc:511] Master 127.3.127.190:32981 requested a full tablet report, sending...
I20260812 02:40:47.165464 30081 raft_consensus.cc:348] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } }
I20260812 02:40:47.165947 30081 raft_consensus.cc:729] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b21083b8daab4989b997fc4d9d660227, State: Initialized, Role: FOLLOWER
I20260812 02:40:47.166450 30081 consensus_queue.cc:260] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } }
I20260812 02:40:47.167425 30081 ts_tablet_manager.cc:1428] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260812 02:40:47.167407 29939 ts_manager.cc:194] Registered new tserver with Master: b21083b8daab4989b997fc4d9d660227 (127.3.127.129:41547)
I20260812 02:40:47.171622 29939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:33909
I20260812 02:40:47.174465 30255 heartbeater.cc:503] Master 127.3.127.190:32981 was elected leader, sending a full tablet report...
Aug 12 02:40:47 dist-test-slave-rmhg krb5kdc[29000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for kudu/127.3.127.129@KRBTEST.COM
I20260812 02:40:47.182588 30268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.170985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:41547 (local address 127.0.0.1:36594)
0812 02:40:47.171419 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:47.171459 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:47.171576 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:47.172285 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:47.172291 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:47.172760 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:47.172917 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:47.172928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.174480 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.174484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:47.175349 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:47.175360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.175999 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.176004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:47.176139 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.176755 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:47.176768 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:47.178496 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:47.180774 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:47.180783 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:47.180788 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:47.181062 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:47.181403 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:47.181406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:47.181407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:47.181492 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:47.182015 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:47.182019 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:47.182152 (+ 133us) client_negotiation.cc:769] Sending connection context
0812 02:40:47.182268 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 02:40:47.182438 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":113,"threads_started":1}
I20260812 02:40:47.182724 30269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.171058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36594 (local address 127.3.127.129:41547)
0812 02:40:47.171840 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:47.171853 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:47.171876 (+ 23us) server_negotiation.cc:407] Connection header received
0812 02:40:47.171945 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:47.171949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:47.172117 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:47.172255 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:47.173070 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.174342 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.175466 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.175876 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.175976 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.178651 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:47.178664 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:47.178680 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:47.178711 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:47.180652 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:47.181204 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:47.181210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:47.181212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:47.181314 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:47.181574 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:47.181577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:47.181578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:47.181897 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:47.182050 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:47.182367 (+ 317us) server_negotiation.cc:299] Negotiation successful
0812 02:40:47.182555 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":145,"threads_started":1}
I20260812 02:40:47.183879 30144 tablet_service.cc:1514] Processing DeleteTablet for tablet 5476e811dff84fd09a14e1f96199fa1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:40:26 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:36594
I20260812 02:40:47.184962 30271 tablet_replica.cc:330] stopping tablet replica
I20260812 02:40:47.185310 30271 raft_consensus.cc:2229] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:40:47.185573 30271 raft_consensus.cc:2258] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:40:47.186599 30271 ts_tablet_manager.cc:1905] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:40:47.190459 30271 ts_tablet_manager.cc:1918] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:40:47.190562 30271 log.cc:1196] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/wal/wals/5476e811dff84fd09a14e1f96199fa1d
I20260812 02:40:47.190794 30271 ts_tablet_manager.cc:1939] T 5476e811dff84fd09a14e1f96199fa1d P b21083b8daab4989b997fc4d9d660227: Deleting consensus metadata
I20260812 02:40:47.192551 29912 catalog_manager.cc:4841] TS b21083b8daab4989b997fc4d9d660227 (127.3.127.129:41547): tablet 5476e811dff84fd09a14e1f96199fa1d (table test-table [id=67025033a82142c5b32987dc598cb950]) successfully deleted
W20260812 02:40:47.282065 30265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:47.282480 30265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:47.282549 30265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:47.286950 30265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:47.287032 30265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:47.287096 30265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:40:47.292743 30265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:39947
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=38711
--tserver_master_addrs=127.3.127.190:32981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:47.293955 30265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:47.294992 30265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:47.305264 30281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:47.305325 30282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:47.305322 30265 server_base.cc:1034] running on GCE node
W20260812 02:40:47.305516 30285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:47.305949 30265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:47.306695 30265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:47.307871 30265 hybrid_clock.cc:648] HybridClock initialized: now 1786502447307849 us; error 35 us; skew 500 ppm
Aug 12 02:40:47 dist-test-slave-rmhg krb5kdc[29000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502447, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:47.310997 30265 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:40:47.312036 30265 webserver.cc:460] Webserver started at http://127.3.127.130:38711/ using document root <none> and password file <none>
I20260812 02:40:47.312630 30265 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:47.312707 30265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:47.317482 30265 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:40:47.319815 30297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:47.321259 30265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:40:47.321388 30265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "019982c5360e4273a895747e6dec7c03"
format_stamp: "Formatted at 2026-08-12 02:40:25 on dist-test-slave-rmhg"
server_key: "eb31442d86214487313216eeda7ad9e5"
server_key_iv: "f7dd39978cbf602c227f6db29625b3f7"
server_key_version: "encryptionkey@0"
I20260812 02:40:47.322001 30265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:47.343768 30265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:47.347818 30265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:47.348004 30265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:47.348637 30265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:40:47.349556 30265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:40:47.349617 30265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:47.349663 30265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:40:47.349682 30265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:47.358774 30265 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:39947
I20260812 02:40:47.358805 30481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:39947 every 8 connection(s)
I20260812 02:40:47.359892 30265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:40:47.360633 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 30265
I20260812 02:40:47.361058 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:37377
--local_ip_for_outbound_sockets=127.3.127.131
--tserver_master_addrs=127.3.127.190:32981
--webserver_port=38583
--webserver_interface=127.3.127.131
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:40:47 dist-test-slave-rmhg krb5kdc[29000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502447, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:40:47.376791 30258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.362131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:36455 (local address 127.3.127.190:32981)
0812 02:40:47.362260 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:47.362264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:47.363730 (+ 1466us) server_negotiation.cc:407] Connection header received
0812 02:40:47.364988 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:47.364992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:47.365069 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:47.365180 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:47.367015 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.367630 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.369537 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.369797 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.369897 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.373001 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:47.373004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:47.373006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:47.373034 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:47.374447 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:47.374974 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:47.374976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:47.374978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:47.375067 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:47.375513 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:47.375515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:47.375516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:47.375749 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:47.375828 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:47.376496 (+ 668us) server_negotiation.cc:299] Negotiation successful
0812 02:40:47.376658 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 02:40:47.377929 30489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.362671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.130:36455)
0812 02:40:47.363533 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:47.363597 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:47.364689 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:47.365413 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:47.365423 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:47.365927 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:47.366738 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:47.366762 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.367790 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.367794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:47.369360 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:47.369375 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.369994 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.370000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:47.370257 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.370938 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:47.370964 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:47.372796 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:47.374573 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:47.374580 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:47.374600 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:47.374869 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:47.375212 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:47.375216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:47.375218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:47.375421 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:47.375844 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:47.375852 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:47.376218 (+ 366us) client_negotiation.cc:769] Sending connection context
0812 02:40:47.376441 (+ 223us) client_negotiation.cc:240] Negotiation successful
0812 02:40:47.376812 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":133,"threads_started":1}
I20260812 02:40:47.379467 30482 heartbeater.cc:347] Connected to a master server at 127.3.127.190:32981
I20260812 02:40:47.379767 30482 heartbeater.cc:464] Registering TS with master...
I20260812 02:40:47.380357 30482 heartbeater.cc:511] Master 127.3.127.190:32981 requested a full tablet report, sending...
I20260812 02:40:47.381542 29939 ts_manager.cc:194] Registered new tserver with Master: 019982c5360e4273a895747e6dec7c03 (127.3.127.130:39947)
I20260812 02:40:47.382151 29939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:36455
W20260812 02:40:47.483697 30488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:40:47.483984 30488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:40:47.484048 30488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:40:47.488329 30488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:40:47.488415 30488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:40:47.488498 30488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:40:47.494204 30488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:37377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=38583
--tserver_master_addrs=127.3.127.190:32981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:40:47.495368 30488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:40:47.496438 30488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:40:47.507068 30501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:47.507155 30488 server_base.cc:1034] running on GCE node
W20260812 02:40:47.507061 30497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:40:47.507303 30498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:40:47.507769 30488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:40:47.508579 30488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:40:47.509750 30488 hybrid_clock.cc:648] HybridClock initialized: now 1786502447509718 us; error 47 us; skew 500 ppm
Aug 12 02:40:47 dist-test-slave-rmhg krb5kdc[29000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502447, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:40:47.512624 30488 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:40:47.513681 30488 webserver.cc:460] Webserver started at http://127.3.127.131:38583/ using document root <none> and password file <none>
I20260812 02:40:47.514286 30488 fs_manager.cc:359] Metadata directory not provided
I20260812 02:40:47.514367 30488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:40:47.518684 30488 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:40:47.521088 30515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:47.522533 30488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:40:47.522691 30488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "af2edfdfdc664199b20e4efdc6656c48"
format_stamp: "Formatted at 2026-08-12 02:40:26 on dist-test-slave-rmhg"
server_key: "ce4a7a8d24f9b731f76976d701f547d1"
server_key_iv: "97b101e642448cc497cd9c959a70a50b"
server_key_version: "encryptionkey@0"
I20260812 02:40:47.523267 30488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:40:47.538269 30488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:40:47.541713 30488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:40:47.541909 30488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:40:47.542543 30488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:40:47.543618 30488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:40:47.543684 30488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:47.543749 30488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:40:47.543781 30488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:40:47.552816 30488 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:37377
I20260812 02:40:47.552843 30714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:37377 every 8 connection(s)
I20260812 02:40:47.553941 30488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:40:47.560570 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 30488
Aug 12 02:40:47 dist-test-slave-rmhg krb5kdc[29000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502447, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:40:47.569061 30258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.556067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:33895 (local address 127.3.127.190:32981)
0812 02:40:47.556262 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:47.556266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:47.557403 (+ 1137us) server_negotiation.cc:407] Connection header received
0812 02:40:47.558323 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:47.558326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:47.558386 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:47.558461 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:47.559688 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.560218 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.561763 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.562101 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.562188 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.564995 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:47.564998 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:47.565001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:47.565022 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:47.566768 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:47.567292 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:47.567295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:47.567297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:47.567378 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:47.567727 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:47.567731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:47.567733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:47.567958 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:47.568066 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:47.568743 (+ 677us) server_negotiation.cc:299] Negotiation successful
0812 02:40:47.568885 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 02:40:47.569453 30720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.561352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38144 (local address 127.3.127.190:32981)
0812 02:40:47.561646 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:47.561650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:47.561776 (+ 126us) server_negotiation.cc:407] Connection header received
0812 02:40:47.561879 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:47.561881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:47.561930 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:47.562028 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:47.562934 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.563547 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.564588 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.564824 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.564933 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.566351 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:47.566355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:47.566358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:47.566400 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:47.567744 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:47.568185 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:47.568189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:47.568190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:47.568289 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:47.568521 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:47.568524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:47.568526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:47.568756 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:47.568995 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:47.569140 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 02:40:47.569294 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"spinlock_wait_cycles":3840,"thread_start_us":98,"threads_started":1}
I20260812 02:40:47.570398 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.556496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.131:33895)
0812 02:40:47.557252 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:47.557297 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:47.558110 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:47.558657 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:47.558664 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:47.559041 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:40:47.559540 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:47.559554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.560389 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.560392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:47.561633 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:47.561646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.562273 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.562278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:47.562491 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.563193 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:40:47.563212 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:40:47.564812 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:40:47.566878 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:47.566883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:47.566898 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:47.567187 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:47.567463 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:40:47.567466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:40:47.567468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:40:47.567632 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:40:47.568071 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:40:47.568077 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:40:47.568436 (+ 359us) client_negotiation.cc:769] Sending connection context
0812 02:40:47.568674 (+ 238us) client_negotiation.cc:240] Negotiation successful
0812 02:40:47.569054 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":119,"threads_started":1}
I20260812 02:40:47.571571 30715 heartbeater.cc:347] Connected to a master server at 127.3.127.190:32981
I20260812 02:40:47.571798 30715 heartbeater.cc:464] Registering TS with master...
I20260812 02:40:47.572294 30715 heartbeater.cc:511] Master 127.3.127.190:32981 requested a full tablet report, sending...
I20260812 02:40:47.573410 29939 ts_manager.cc:194] Registered new tserver with Master: af2edfdfdc664199b20e4efdc6656c48 (127.3.127.131:37377)
I20260812 02:40:47.574039 29939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:33895
I20260812 02:40:47.583513 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:40:47.588194 30720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.584275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.3.127.190:32981)
0812 02:40:47.584544 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:47.584548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:47.584680 (+ 132us) server_negotiation.cc:407] Connection header received
0812 02:40:47.584901 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:47.584903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:47.585029 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:47.585132 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:40:47.585427 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.585972 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.586873 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.587122 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.587219 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.587587 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:40:47.587748 (+ 161us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0812 02:40:47.588049 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38154: token expired
Metrics: {"server-negotiator.queue_time_us":96}
W20260812 02:40:47.588279 30720 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38154: token expired
W20260812 02:40:47.588691 29827 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.127.190:32981)
I20260812 02:40:47.596146 30720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.589120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.3.127.190:32981)
0812 02:40:47.589272 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:47.589275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:47.589288 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:40:47.589378 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:47.589380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:47.589422 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:47.589484 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:40:47.590111 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.590597 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.591398 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.591669 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.591760 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.593084 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:40:47.593088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:40:47.593090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:40:47.593116 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:40:47.594603 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:47.595042 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:47.595045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:47.595046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:47.595120 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:40:47.595323 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:40:47.595325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:40:47.595326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:40:47.595552 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:40:47.595658 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:47.595888 (+ 230us) server_negotiation.cc:299] Negotiation successful
0812 02:40:47.596014 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 02:40:47.598958 29939 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:40:47.602654 29939 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:40:47.611104 30144 tablet_service.cc:1467] Processing CreateTablet for tablet a877ae3e8700493b89a336e69bf1fe3e (DEFAULT_TABLE table=test-table [id=9a2f9a015d514081bec26dd7adefe379]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:40:47.611837 30144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a877ae3e8700493b89a336e69bf1fe3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:40:47.614792 30081 tablet_bootstrap.cc:492] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227: Bootstrap starting.
I20260812 02:40:47.615113 30268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.609781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:37377 (local address 127.0.0.1:50984)
0812 02:40:47.609905 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:47.609921 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:47.610006 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:47.611031 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:47.611035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:47.611055 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:40:47.611157 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:47.611161 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.612526 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.612530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:47.613445 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:47.613459 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.614630 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.614634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:47.614763 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.614823 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 02:40:47.614879 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 02:40:47.614967 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 02:40:47.615732 30081 tablet_bootstrap.cc:654] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227: Neither blocks nor log segments found. Creating new log.
I20260812 02:40:47.615785 30728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.609979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.3.127.131:37377)
0812 02:40:47.610413 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:47.610425 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:47.610451 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:40:47.610550 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:47.610556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:47.610830 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:47.611050 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:40:47.611235 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.612393 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.613595 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.614493 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.615297 (+ 804us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.615470 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:47.615526 (+ 56us) server_negotiation.cc:299] Negotiation successful
0812 02:40:47.615629 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":208,"threads_started":1}
I20260812 02:40:47.617331 30081 tablet_bootstrap.cc:492] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227: No bootstrap required, opened a new log
I20260812 02:40:47.617427 30081 ts_tablet_manager.cc:1397] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:40:47.617920 30081 raft_consensus.cc:348] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } }
I20260812 02:40:47.618052 30081 raft_consensus.cc:374] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:40:47.618101 30081 raft_consensus.cc:729] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b21083b8daab4989b997fc4d9d660227, State: Initialized, Role: FOLLOWER
I20260812 02:40:47.618268 30081 consensus_queue.cc:260] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } }
I20260812 02:40:47.618646 30081 ts_tablet_manager.cc:1428] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:40:47.619298 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.610659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.3.127.130:39947)
0812 02:40:47.611003 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:47.611016 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:47.613476 (+ 2460us) server_negotiation.cc:407] Connection header received
0812 02:40:47.613621 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:47.613627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:47.613875 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:47.614041 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:40:47.614242 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.615304 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.617590 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.618438 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.618541 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.618696 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:47.619044 (+ 348us) server_negotiation.cc:299] Negotiation successful
0812 02:40:47.619153 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":102,"threads_started":1}
I20260812 02:40:47.618873 30604 tablet_service.cc:1467] Processing CreateTablet for tablet a877ae3e8700493b89a336e69bf1fe3e (DEFAULT_TABLE table=test-table [id=9a2f9a015d514081bec26dd7adefe379]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:40:47.620004 30604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a877ae3e8700493b89a336e69bf1fe3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:40:47.620002 30729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.610218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:39947 (local address 127.0.0.1:51868)
0812 02:40:47.613344 (+ 3126us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:47.613359 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:47.613474 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:47.614041 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:47.614044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:47.614065 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:40:47.614162 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:47.614168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.615654 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.615658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:47.617436 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:47.617448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.618769 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.618774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:47.618888 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.618914 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 02:40:47.618990 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 02:40:47.619071 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3052,"thread_start_us":124,"threads_started":1}
I20260812 02:40:47.622817 30377 tablet_service.cc:1467] Processing CreateTablet for tablet a877ae3e8700493b89a336e69bf1fe3e (DEFAULT_TABLE table=test-table [id=9a2f9a015d514081bec26dd7adefe379]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:40:47.623773 30377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a877ae3e8700493b89a336e69bf1fe3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:40:47.627502 30734 tablet_bootstrap.cc:492] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48: Bootstrap starting.
I20260812 02:40:47.630615 30734 tablet_bootstrap.cc:654] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48: Neither blocks nor log segments found. Creating new log.
I20260812 02:40:47.631611 30734 log.cc:824] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48: Log is configured to *not* fsync() on all Append() calls
I20260812 02:40:47.631976 30735 tablet_bootstrap.cc:492] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03: Bootstrap starting.
I20260812 02:40:47.633875 30734 tablet_bootstrap.cc:492] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48: No bootstrap required, opened a new log
I20260812 02:40:47.634070 30734 ts_tablet_manager.cc:1397] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 02:40:47.634713 30735 tablet_bootstrap.cc:654] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03: Neither blocks nor log segments found. Creating new log.
I20260812 02:40:47.635610 30735 log.cc:824] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03: Log is configured to *not* fsync() on all Append() calls
I20260812 02:40:47.637863 30735 tablet_bootstrap.cc:492] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03: No bootstrap required, opened a new log
I20260812 02:40:47.637789 30734 raft_consensus.cc:348] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } }
I20260812 02:40:47.638084 30735 ts_tablet_manager.cc:1397] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 02:40:47.638078 30734 raft_consensus.cc:374] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:40:47.638140 30734 raft_consensus.cc:729] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af2edfdfdc664199b20e4efdc6656c48, State: Initialized, Role: FOLLOWER
I20260812 02:40:47.638608 30734 consensus_queue.cc:260] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } }
I20260812 02:40:47.639451 30715 heartbeater.cc:503] Master 127.3.127.190:32981 was elected leader, sending a full tablet report...
I20260812 02:40:47.639745 30734 ts_tablet_manager.cc:1428] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 02:40:47.642063 30735 raft_consensus.cc:348] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } }
I20260812 02:40:47.642409 30735 raft_consensus.cc:374] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:40:47.642479 30735 raft_consensus.cc:729] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 019982c5360e4273a895747e6dec7c03, State: Initialized, Role: FOLLOWER
I20260812 02:40:47.642977 30735 consensus_queue.cc:260] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } }
W20260812 02:40:47.643101 30257 tablet.cc:2369] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:40:47.643906 30482 heartbeater.cc:503] Master 127.3.127.190:32981 was elected leader, sending a full tablet report...
I20260812 02:40:47.644129 30735 ts_tablet_manager.cc:1428] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 02:40:47.650187 30742 raft_consensus.cc:482] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:40:47.650381 30742 raft_consensus.cc:504] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } }
I20260812 02:40:47.651528 30742 leader_election.cc:288] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af2edfdfdc664199b20e4efdc6656c48 (127.3.127.131:37377), b21083b8daab4989b997fc4d9d660227 (127.3.127.129:41547)
I20260812 02:40:47.653680 30266 raft_consensus.cc:482] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:40:47.653884 30266 raft_consensus.cc:504] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } }
I20260812 02:40:47.655058 30266 leader_election.cc:288] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af2edfdfdc664199b20e4efdc6656c48 (127.3.127.131:37377), 019982c5360e4273a895747e6dec7c03 (127.3.127.130:39947)
I20260812 02:40:47.656006 30489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.651821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:41547 (local address 127.3.127.130:46277)
0812 02:40:47.652001 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:47.652015 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:47.652101 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:47.652363 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:47.652366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:47.652385 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:40:47.652541 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:47.652547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.653380 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.653383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:47.654364 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:47.654375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.655503 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.655507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:47.655662 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.655705 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 02:40:47.655777 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 02:40:47.655856 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":8}
I20260812 02:40:47.656018 30728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.651880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:44105 (local address 127.3.127.131:37377)
0812 02:40:47.652008 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:47.652013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:47.652319 (+ 306us) server_negotiation.cc:407] Connection header received
0812 02:40:47.652461 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:47.652464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:47.652509 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:47.652573 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:40:47.652820 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.653365 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.654685 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.655283 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.655391 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.655437 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:47.655831 (+ 394us) server_negotiation.cc:299] Negotiation successful
0812 02:40:47.655914 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 02:40:47.656495 30269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.651932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:46277 (local address 127.3.127.129:41547)
0812 02:40:47.652107 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:47.652112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:47.652128 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:40:47.652184 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:47.652186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:47.652238 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:47.652333 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:40:47.652670 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.653272 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.654493 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.655105 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.655226 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.655377 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:47.656299 (+ 922us) server_negotiation.cc:299] Negotiation successful
0812 02:40:47.656382 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260812 02:40:47.657012 30177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a877ae3e8700493b89a336e69bf1fe3e" candidate_uuid: "019982c5360e4273a895747e6dec7c03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b21083b8daab4989b997fc4d9d660227" is_pre_election: true
I20260812 02:40:47.657315 30177 raft_consensus.cc:2454] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 019982c5360e4273a895747e6dec7c03 in term 0.
I20260812 02:40:47.657979 30310 leader_election.cc:302] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019982c5360e4273a895747e6dec7c03, b21083b8daab4989b997fc4d9d660227; no voters:
I20260812 02:40:47.658285 30742 raft_consensus.cc:2781] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:40:47.658394 30742 raft_consensus.cc:482] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:40:47.658450 30742 raft_consensus.cc:3037] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:40:47.658681 30744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.651830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:37377 (local address 127.3.127.130:44105)
0812 02:40:47.652233 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:47.652243 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:47.652337 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:47.652590 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:47.652592 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:47.652608 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:40:47.652718 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:47.652723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.653475 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.653478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:47.654542 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:47.654553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.655503 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.655507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:47.655685 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.655711 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 02:40:47.656420 (+ 709us) client_negotiation.cc:240] Negotiation successful
0812 02:40:47.656511 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":96,"thread_start_us":139,"threads_started":1}
I20260812 02:40:47.659293 30636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a877ae3e8700493b89a336e69bf1fe3e" candidate_uuid: "019982c5360e4273a895747e6dec7c03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af2edfdfdc664199b20e4efdc6656c48" is_pre_election: true
I20260812 02:40:47.659621 30636 raft_consensus.cc:2454] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 019982c5360e4273a895747e6dec7c03 in term 0.
I20260812 02:40:47.659785 30742 raft_consensus.cc:504] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } }
I20260812 02:40:47.660152 30742 leader_election.cc:288] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [CANDIDATE]: Term 1 election: Requested vote from peers af2edfdfdc664199b20e4efdc6656c48 (127.3.127.131:37377), b21083b8daab4989b997fc4d9d660227 (127.3.127.129:41547)
I20260812 02:40:47.660427 30260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.655234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:37377 (local address 127.3.127.129:36497)
0812 02:40:47.655339 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:47.655353 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:47.655433 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:47.656395 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:47.656398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:47.656420 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:40:47.656509 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:47.656515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.657382 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.657385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:47.658480 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:47.658493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.659924 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.659928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:47.660029 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.660112 (+ 83us) client_negotiation.cc:769] Sending connection context
0812 02:40:47.660195 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 02:40:47.660276 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":30}
I20260812 02:40:47.660564 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.656293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:35781 (local address 127.3.127.130:39947)
0812 02:40:47.656394 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:47.656398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:47.656409 (+ 11us) server_negotiation.cc:407] Connection header received
0812 02:40:47.656443 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:47.656446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:47.656485 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:47.656552 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:40:47.657369 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.657912 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.659268 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.659802 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.659888 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.659932 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:47.660371 (+ 439us) server_negotiation.cc:299] Negotiation successful
0812 02:40:47.660444 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 02:40:47.660669 30177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a877ae3e8700493b89a336e69bf1fe3e" candidate_uuid: "019982c5360e4273a895747e6dec7c03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b21083b8daab4989b997fc4d9d660227"
I20260812 02:40:47.660753 30636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a877ae3e8700493b89a336e69bf1fe3e" candidate_uuid: "019982c5360e4273a895747e6dec7c03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af2edfdfdc664199b20e4efdc6656c48"
I20260812 02:40:47.660792 30177 raft_consensus.cc:3037] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:40:47.660936 30746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.655505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:39947 (local address 127.3.127.129:35781)
0812 02:40:47.655797 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:47.655810 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:40:47.655894 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:40:47.657003 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:40:47.657006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:40:47.657022 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:40:47.657137 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:47.657241 (+ 104us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.658066 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.658069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:47.659127 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:40:47.659138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.660041 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.660045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:40:47.660225 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.660252 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 02:40:47.660690 (+ 438us) client_negotiation.cc:240] Negotiation successful
0812 02:40:47.660767 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":27,"thread_start_us":93,"threads_started":1}
I20260812 02:40:47.660863 30636 raft_consensus.cc:3037] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:40:47.661454 30728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.655934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:36497 (local address 127.3.127.131:37377)
0812 02:40:47.656177 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:47.656181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:47.656194 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:40:47.656241 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:47.656243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:47.656289 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:47.656347 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:40:47.656625 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.657256 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.658591 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.659125 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.659231 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.659278 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:47.660779 (+ 1501us) server_negotiation.cc:299] Negotiation successful
0812 02:40:47.660852 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":135,"threads_started":1}
I20260812 02:40:47.661574 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a877ae3e8700493b89a336e69bf1fe3e" candidate_uuid: "b21083b8daab4989b997fc4d9d660227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "019982c5360e4273a895747e6dec7c03" is_pre_election: true
I20260812 02:40:47.661803 30408 raft_consensus.cc:2379] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b21083b8daab4989b997fc4d9d660227 in current term 1: Already voted for candidate 019982c5360e4273a895747e6dec7c03 in this term.
I20260812 02:40:47.662031 30177 raft_consensus.cc:2454] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 019982c5360e4273a895747e6dec7c03 in term 1.
I20260812 02:40:47.662235 30636 raft_consensus.cc:2454] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 019982c5360e4273a895747e6dec7c03 in term 1.
I20260812 02:40:47.662334 30310 leader_election.cc:302] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019982c5360e4273a895747e6dec7c03, b21083b8daab4989b997fc4d9d660227; no voters:
I20260812 02:40:47.662438 30634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a877ae3e8700493b89a336e69bf1fe3e" candidate_uuid: "b21083b8daab4989b997fc4d9d660227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af2edfdfdc664199b20e4efdc6656c48" is_pre_election: true
I20260812 02:40:47.662595 30634 raft_consensus.cc:2379] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b21083b8daab4989b997fc4d9d660227 in current term 1: Already voted for candidate 019982c5360e4273a895747e6dec7c03 in this term.
I20260812 02:40:47.662885 30742 raft_consensus.cc:2781] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:40:47.663033 30075 leader_election.cc:302] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b21083b8daab4989b997fc4d9d660227; no voters: 019982c5360e4273a895747e6dec7c03, af2edfdfdc664199b20e4efdc6656c48
I20260812 02:40:47.663133 30742 raft_consensus.cc:686] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [term 1 LEADER]: Becoming Leader. State: Replica: 019982c5360e4273a895747e6dec7c03, State: Running, Role: LEADER
I20260812 02:40:47.663498 30266 raft_consensus.cc:2726] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 02:40:47.663484 30742 consensus_queue.cc:237] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } }
I20260812 02:40:47.666802 29937 catalog_manager.cc:5495] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 019982c5360e4273a895747e6dec7c03 (127.3.127.130). New cstate: current_term: 1 leader_uuid: "019982c5360e4273a895747e6dec7c03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } health_report { overall_health: UNKNOWN } } }
I20260812 02:40:47.689131 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.685576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51872 (local address 127.3.127.130:39947)
0812 02:40:47.685742 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 02:40:47.685747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:40:47.685760 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:40:47.685842 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:40:47.685844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:40:47.685899 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:40:47.685987 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:40:47.686251 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.686762 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.687525 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:40:47.687741 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:40:47.687807 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:40:47.688038 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:40:47.688472 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:40:47.688562 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 02:40:47.688645 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 02:40:47.688879 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 02:40:47.695472 30634 raft_consensus.cc:1261] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48 [term 1 FOLLOWER]: Refusing update from remote peer 019982c5360e4273a895747e6dec7c03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:40:47.695518 30177 raft_consensus.cc:1261] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Refusing update from remote peer 019982c5360e4273a895747e6dec7c03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:40:47.696158 30752 consensus_queue.cc:1035] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:40:47.696367 30742 consensus_queue.cc:1035] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:40:47.703382 30760 mvcc.cc:205] Tried to move back new op lower bound from 7317514025753022464 to 7317514025632423936. Current Snapshot: MvccSnapshot[applied={T|T < 7317514025753022464}]
I20260812 02:40:47.713265 29939 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:40:47.713452 29939 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:40:47.716034 29939 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd: Sending DeleteTablet for 3 replicas of tablet a877ae3e8700493b89a336e69bf1fe3e
I20260812 02:40:47.716871 30377 tablet_service.cc:1514] Processing DeleteTablet for tablet a877ae3e8700493b89a336e69bf1fe3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:40:47 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:51868
I20260812 02:40:47.716874 30604 tablet_service.cc:1514] Processing DeleteTablet for tablet a877ae3e8700493b89a336e69bf1fe3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:40:47 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:50984
I20260812 02:40:47.716943 30144 tablet_service.cc:1514] Processing DeleteTablet for tablet a877ae3e8700493b89a336e69bf1fe3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:40:47 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:36594
I20260812 02:40:47.717427 30774 tablet_replica.cc:330] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227: stopping tablet replica
I20260812 02:40:47.717571 30774 raft_consensus.cc:2229] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:40:47.717710 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 30005
I20260812 02:40:47.717736 30774 raft_consensus.cc:2258] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:40:47.730123 30775 tablet_replica.cc:330] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03: stopping tablet replica
I20260812 02:40:47.730579 30775 raft_consensus.cc:2229] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:40:47.731057 30775 raft_consensus.cc:2258] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:40:47.733690 30776 tablet_replica.cc:330] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48: stopping tablet replica
I20260812 02:40:47.734160 30776 raft_consensus.cc:2229] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:40:47.734320 30775 ts_tablet_manager.cc:1905] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:40:47.734638 30776 raft_consensus.cc:2258] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:40:47.736299 30776 ts_tablet_manager.cc:1905] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:40:47.737260 30775 ts_tablet_manager.cc:1918] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:40:47.737363 30775 log.cc:1196] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/a877ae3e8700493b89a336e69bf1fe3e
I20260812 02:40:47.737687 30775 ts_tablet_manager.cc:1939] T a877ae3e8700493b89a336e69bf1fe3e P 019982c5360e4273a895747e6dec7c03: Deleting consensus metadata
I20260812 02:40:47.739313 30776 ts_tablet_manager.cc:1918] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:40:47.739329 29910 catalog_manager.cc:4841] TS 019982c5360e4273a895747e6dec7c03 (127.3.127.130:39947): tablet a877ae3e8700493b89a336e69bf1fe3e (table test-table [id=9a2f9a015d514081bec26dd7adefe379]) successfully deleted
I20260812 02:40:47.739514 30776 log.cc:1196] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/a877ae3e8700493b89a336e69bf1fe3e
I20260812 02:40:47.739917 30776 ts_tablet_manager.cc:1939] T a877ae3e8700493b89a336e69bf1fe3e P af2edfdfdc664199b20e4efdc6656c48: Deleting consensus metadata
I20260812 02:40:47.741209 29913 catalog_manager.cc:4841] TS af2edfdfdc664199b20e4efdc6656c48 (127.3.127.131:37377): tablet a877ae3e8700493b89a336e69bf1fe3e (table test-table [id=9a2f9a015d514081bec26dd7adefe379]) successfully deleted
W20260812 02:40:47.741279 29912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.127.129:41547 (error 108)
I20260812 02:40:47.742317 30729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:40:47.742030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:41547 (local address 127.0.0.1:36600)
0812 02:40:47.742163 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 02:40:47.742232 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.129:41547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260812 02:40:47.742537 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 30265
W20260812 02:40:47.742537 29912 catalog_manager.cc:4584] TS b21083b8daab4989b997fc4d9d660227 (127.3.127.129:41547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a877ae3e8700493b89a336e69bf1fe3e: Network error: Client connection negotiation failed: client connection to 127.3.127.129:41547: connect: Connection refused (error 111)
I20260812 02:40:47.758476 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 30488
I20260812 02:40:47.774300 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 29881
I20260812 02:41:07.792114 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:32981
--webserver_interface=127.3.127.190
--webserver_port=40603
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:32981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:07.911604 30779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:07.911886 30779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:07.911927 30779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:07.916143 30779 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 02:41:07.916296 30779 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 02:41:07.916332 30779 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 02:41:07.916347 30779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:07.916365 30779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:07.916386 30779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:07.916402 30779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:07.916421 30779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:07.921995 30779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:32981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:32981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=40603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:07.923156 30779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:07.924181 30779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:07.933290 30794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:07.933313 30791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:07.933298 30790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:07.933357 30779 server_base.cc:1034] running on GCE node
I20260812 02:41:07.934101 30779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:07.935298 30779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:07.936450 30779 hybrid_clock.cc:648] HybridClock initialized: now 1786502467936430 us; error 36 us; skew 500 ppm
Aug 12 02:41:07 dist-test-slave-rmhg krb5kdc[29000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502467, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:07.942885 30779 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:41:07.943974 30779 webserver.cc:460] Webserver started at http://127.3.127.190:40603/ using document root <none> and password file <none>
I20260812 02:41:07.944592 30779 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:07.944659 30779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:07.948942 30779 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:41:07.951438 30806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:07.952924 30779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:41:07.953081 30779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "6b4695b32eda45029e436c362b9ffacd"
format_stamp: "Formatted at 2026-08-12 02:40:25 on dist-test-slave-rmhg"
server_key: "c7ba7395d0e77710c909535a73bb30c8"
server_key_iv: "89031a75abfc5700c3108d2bfdf30177"
server_key_version: "encryptionkey@0"
I20260812 02:41:07.953766 30779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:07.969494 30779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:07.973117 30779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:07.973371 30779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:07.981061 30779 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:32981
I20260812 02:41:07.981375 30891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:32981 every 8 connection(s)
I20260812 02:41:07.982487 30779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:07.987290 30894 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:41:07.989125 30894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd: Bootstrap starting.
I20260812 02:41:07.991214 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 30779
I20260812 02:41:07.991758 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:41547
--local_ip_for_outbound_sockets=127.3.127.129
--tserver_master_addrs=127.3.127.190:32981
--webserver_port=45345
--webserver_interface=127.3.127.129
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:41:08.008670 30894 log.cc:824] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:08.020709 30894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:41:08.021103 30894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd: Bootstrap complete.
I20260812 02:41:08.025274 30894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4695b32eda45029e436c362b9ffacd" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 32981 } }
I20260812 02:41:08.025681 30894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4695b32eda45029e436c362b9ffacd, State: Initialized, Role: FOLLOWER
I20260812 02:41:08.026225 30894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4695b32eda45029e436c362b9ffacd" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 32981 } }
I20260812 02:41:08.026353 30894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:08.026419 30894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:08.026515 30894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 2 FOLLOWER]: Advancing to term 3
I20260812 02:41:08.028486 30894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4695b32eda45029e436c362b9ffacd" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 32981 } }
I20260812 02:41:08.028811 30894 leader_election.cc:302] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b4695b32eda45029e436c362b9ffacd; no voters:
I20260812 02:41:08.029119 30894 leader_election.cc:288] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 02:41:08.029291 30900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 3 FOLLOWER]: Leader election won for term 3
I20260812 02:41:08.029546 30900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [term 3 LEADER]: Becoming Leader. State: Replica: 6b4695b32eda45029e436c362b9ffacd, State: Running, Role: LEADER
I20260812 02:41:08.029955 30900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4695b32eda45029e436c362b9ffacd" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 32981 } }
I20260812 02:41:08.030386 30894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:08.031471 30902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6b4695b32eda45029e436c362b9ffacd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4695b32eda45029e436c362b9ffacd" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 32981 } } }
I20260812 02:41:08.031615 30902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:08.032290 30915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:08.032554 30903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b4695b32eda45029e436c362b9ffacd. Latest consensus state: current_term: 3 leader_uuid: "6b4695b32eda45029e436c362b9ffacd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4695b32eda45029e436c362b9ffacd" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 32981 } } }
I20260812 02:41:08.032672 30903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:08.036685 30915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:08.037302 30915 catalog_manager.cc:1259] Loaded cluster ID: bc11ec4797f64b9d9ab4e65907b3988d
I20260812 02:41:08.037351 30915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:08.039188 30915 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:08.039836 30915 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd: Loaded TSK: 0
I20260812 02:41:08.039906 30915 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6b4695b32eda45029e436c362b9ffacd: Loaded TSK: 1
F20260812 02:41:08.040547 30915 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe3303a09ad google::LogMessage::Fail() at ??:0
@ 0x7fe3303a388a google::LogMessage::SendToLog() at ??:0
@ 0x7fe3303a04de google::LogMessage::Flush() at ??:0
@ 0x7fe3303a1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe33306418d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe33305cf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe33ad057bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe33acdf99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe33ace0178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fe33ad21df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe33ad4082b std::function<>::operator()() at ??:0
@ 0x7fe33acdfb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe33ace0f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe33acdd579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe33ad21640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe3325f23d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe3325f2c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe3325f43f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe3325e3512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe333c276db start_thread at ??:0
@ 0x7fe32ee9e61f clone at ??:0
W20260812 02:41:08.137637 30898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:08.137894 30898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:08.137938 30898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:08.142362 30898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:08.142418 30898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:08.142480 30898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:41:08.148315 30898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:41547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=45345
--tserver_master_addrs=127.3.127.190:32981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:08.149681 30898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:08.150691 30898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:08.161027 30976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:08.160986 30975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:08.161248 30898 server_base.cc:1034] running on GCE node
W20260812 02:41:08.161377 30979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:08.161924 30898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:08.162804 30898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:08.164005 30898 hybrid_clock.cc:648] HybridClock initialized: now 1786502468163971 us; error 49 us; skew 500 ppm
Aug 12 02:41:08 dist-test-slave-rmhg krb5kdc[29000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502468, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:08.167737 30898 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:41:08.169052 30898 webserver.cc:460] Webserver started at http://127.3.127.129:45345/ using document root <none> and password file <none>
I20260812 02:41:08.169907 30898 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:08.170011 30898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:08.175688 30898 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:41:08.178763 30989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:08.180581 30898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260812 02:41:08.180747 30898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "b21083b8daab4989b997fc4d9d660227"
format_stamp: "Formatted at 2026-08-12 02:40:25 on dist-test-slave-rmhg"
server_key: "5d1556eb453495c340acd458b8d6206f"
server_key_iv: "19e438a457cebe867f6f5fadd528dc74"
server_key_version: "encryptionkey@0"
I20260812 02:41:08.181614 30898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:08.216403 30898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:08.221220 30898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:08.221460 30898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:08.222291 30898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:41:08.223883 31002 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 02:41:08.228482 30898 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 02:41:08.228587 30898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260812 02:41:08.228674 30898 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 02:41:08.231199 30898 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 02:41:08.231278 30898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260812 02:41:08.231594 31002 tablet_bootstrap.cc:492] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227: Bootstrap starting.
I20260812 02:41:08.242914 30898 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:41547
I20260812 02:41:08.243306 31191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:41547 every 8 connection(s)
I20260812 02:41:08.244292 30898 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:41:08.250941 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 30898
I20260812 02:41:08.251484 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:39947
--local_ip_for_outbound_sockets=127.3.127.130
--tserver_master_addrs=127.3.127.190:32981
--webserver_port=38711
--webserver_interface=127.3.127.130
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:41:08.252753 31002 log.cc:824] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:08.262782 31002 tablet_bootstrap.cc:492] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:41:08.263293 31002 tablet_bootstrap.cc:492] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227: Bootstrap complete.
I20260812 02:41:08.264925 31002 ts_tablet_manager.cc:1397] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.004s
I20260812 02:41:08.268627 31002 raft_consensus.cc:348] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } }
I20260812 02:41:08.269311 31002 raft_consensus.cc:729] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b21083b8daab4989b997fc4d9d660227, State: Initialized, Role: FOLLOWER
I20260812 02:41:08.269958 31002 consensus_queue.cc:260] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } }
I20260812 02:41:08.271238 31002 ts_tablet_manager.cc:1428] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227: Time spent starting tablet: real 0.006s user 0.006s sys 0.003s
W20260812 02:41:08.322183 31195 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:41:08.247067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.129:37059)
0812 02:41:08.247766 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:08.247841 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:08.249066 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:08.320967 (+ 71901us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":178,"threads_started":1}
W20260812 02:41:08.322757 31193 heartbeater.cc:644] Failed to heartbeat to 127.3.127.190:32981 (0 consecutive failures): Network error: Failed to ping master at 127.3.127.190:32981: Client connection negotiation failed: client connection to 127.3.127.190:32981: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 02:41:08.323742 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:08.323410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.129:56743)
0812 02:41:08.323567 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:08.323660 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260812 02:41:08.324545 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:08.324305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.129:37679)
0812 02:41:08.324416 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:08.324481 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 02:41:08.324753 31193 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 02:41:08.378863 31198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:08.379268 31198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:08.379338 31198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:08.385042 31198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:08.385129 31198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:08.385242 31198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:41:08.391415 31198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:39947
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=38711
--tserver_master_addrs=127.3.127.190:32981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:08.392656 31198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:08.393832 31198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:08.404621 31211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:08.404781 31198 server_base.cc:1034] running on GCE node
W20260812 02:41:08.404634 31207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:08.405036 31213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:08.405520 31198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:08.406237 31198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:08.407404 31198 hybrid_clock.cc:648] HybridClock initialized: now 1786502468407366 us; error 61 us; skew 500 ppm
Aug 12 02:41:08 dist-test-slave-rmhg krb5kdc[29000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502468, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:08.410377 31198 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:41:08.411331 31198 webserver.cc:460] Webserver started at http://127.3.127.130:38711/ using document root <none> and password file <none>
I20260812 02:41:08.411916 31198 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:08.411984 31198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:08.417055 31198 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260812 02:41:08.419445 31226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:08.420763 31198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 02:41:08.420887 31198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "019982c5360e4273a895747e6dec7c03"
format_stamp: "Formatted at 2026-08-12 02:40:25 on dist-test-slave-rmhg"
server_key: "eb31442d86214487313216eeda7ad9e5"
server_key_iv: "f7dd39978cbf602c227f6db29625b3f7"
server_key_version: "encryptionkey@0"
I20260812 02:41:08.421530 31198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:08.444392 31198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:08.447962 31198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:08.448189 31198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:08.448844 31198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:41:08.449800 31198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:41:08.449852 31198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:08.449908 31198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:41:08.449937 31198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:08.459008 31198 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:39947
I20260812 02:41:08.459053 31413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:39947 every 8 connection(s)
I20260812 02:41:08.460084 31198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:41:08.463956 31421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:08.462353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.130:47681)
0812 02:41:08.463033 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:08.463183 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":124,"threads_started":1}
W20260812 02:41:08.464479 31416 heartbeater.cc:644] Failed to heartbeat to 127.3.127.190:32981 (0 consecutive failures): Network error: Failed to ping master at 127.3.127.190:32981: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
I20260812 02:41:08.465135 31421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:08.464911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.130:49217)
0812 02:41:08.465023 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:08.465076 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260812 02:41:08.465827 31421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:08.465628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.130:38885)
0812 02:41:08.465733 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:08.465777 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 02:41:08.465991 31416 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 02:41:08.467558 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 31198
I20260812 02:41:08.467967 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:37377
--local_ip_for_outbound_sockets=127.3.127.131
--tserver_master_addrs=127.3.127.190:32981
--webserver_port=38583
--webserver_interface=127.3.127.131
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:08.588603 31422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:08.588902 31422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:08.588948 31422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:08.593080 31422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:08.593194 31422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:08.593278 31422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:41:08.598958 31422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:37377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=38583
--tserver_master_addrs=127.3.127.190:32981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:08.600154 31422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:08.601432 31422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:08.612102 31430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:08.612102 31437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:08.612103 31432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:08.612495 31422 server_base.cc:1034] running on GCE node
I20260812 02:41:08.612995 31422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:08.613843 31422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:08.615031 31422 hybrid_clock.cc:648] HybridClock initialized: now 1786502468614980 us; error 63 us; skew 500 ppm
Aug 12 02:41:08 dist-test-slave-rmhg krb5kdc[29000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502468, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:08.618242 31422 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:41:08.619395 31422 webserver.cc:460] Webserver started at http://127.3.127.131:38583/ using document root <none> and password file <none>
I20260812 02:41:08.620002 31422 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:08.620078 31422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:08.624339 31422 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:41:08.626781 31447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:08.628233 31422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:41:08.628362 31422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "af2edfdfdc664199b20e4efdc6656c48"
format_stamp: "Formatted at 2026-08-12 02:40:26 on dist-test-slave-rmhg"
server_key: "ce4a7a8d24f9b731f76976d701f547d1"
server_key_iv: "97b101e642448cc497cd9c959a70a50b"
server_key_version: "encryptionkey@0"
I20260812 02:41:08.628943 31422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:08.661001 31422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:08.664381 31422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:08.664609 31422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:08.665293 31422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:41:08.666230 31422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:41:08.666282 31422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:08.666340 31422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:41:08.666366 31422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:08.675819 31422 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:37377
I20260812 02:41:08.675849 31635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:37377 every 8 connection(s)
I20260812 02:41:08.676986 31422 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:41:08.681288 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:08.679495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.131:52799)
0812 02:41:08.680242 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:08.680365 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":117,"threads_started":1}
W20260812 02:41:08.681855 31637 heartbeater.cc:644] Failed to heartbeat to 127.3.127.190:32981 (0 consecutive failures): Network error: Failed to ping master at 127.3.127.190:32981: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
I20260812 02:41:08.682618 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:08.682356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.131:58469)
0812 02:41:08.682489 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:08.682551 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260812 02:41:08.683369 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:08.683169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.131:49777)
0812 02:41:08.683270 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:08.683316 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260812 02:41:08.683566 31637 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 02:41:08.686530 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 31422
I20260812 02:41:09.326074 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:09.325523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.129:39595)
0812 02:41:09.325871 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:09.325943 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":111,"threads_started":1}
I20260812 02:41:09.467084 31644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:09.466572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.130:53851)
0812 02:41:09.466894 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:09.466957 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":119,"threads_started":1}
I20260812 02:41:09.509390 31645 raft_consensus.cc:482] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:41:09.509657 31645 raft_consensus.cc:504] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } }
I20260812 02:41:09.511001 31645 leader_election.cc:288] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af2edfdfdc664199b20e4efdc6656c48 (127.3.127.131:37377), 019982c5360e4273a895747e6dec7c03 (127.3.127.130:39947)
Aug 12 02:41:09 dist-test-slave-rmhg krb5kdc[29000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502468, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.131@KRBTEST.COM
Aug 12 02:41:09 dist-test-slave-rmhg krb5kdc[29000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502468, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.130@KRBTEST.COM
I20260812 02:41:09.525475 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:09.511278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:37377 (local address 127.3.127.129:43365)
0812 02:41:09.511387 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:09.511402 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:09.511507 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:09.513458 (+ 1951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:09.513464 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:09.514022 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:41:09.514551 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:09.514567 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:09.515927 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:09.515930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:09.516935 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:09.516946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:09.517749 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:09.517753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:09.517987 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:09.518650 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:41:09.518662 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:41:09.520330 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:41:09.522548 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:09.522555 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:09.522571 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:09.522855 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:09.523230 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:09.523233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:09.523234 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:09.523384 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:09.524240 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:41:09.524246 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:41:09.524633 (+ 387us) client_negotiation.cc:769] Sending connection context
0812 02:41:09.524872 (+ 239us) client_negotiation.cc:240] Negotiation successful
0812 02:41:09.525284 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":174,"spinlock_wait_cycles":32896}
I20260812 02:41:09.525597 31647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:09.511483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:55873 (local address 127.3.127.130:39947)
0812 02:41:09.511934 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:09.511955 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:09.511989 (+ 34us) server_negotiation.cc:407] Connection header received
0812 02:41:09.512734 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:09.512741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:09.513055 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:09.513373 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:09.514736 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:09.515763 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:09.517066 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:09.517617 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:09.517784 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:09.520671 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:09.520676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:09.520689 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:09.520718 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:09.522748 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:09.523175 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:09.523182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:09.523187 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:09.523321 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:09.523594 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:09.523598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:09.523599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:09.524138 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:09.524255 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:09.525033 (+ 778us) server_negotiation.cc:299] Negotiation successful
0812 02:41:09.525407 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":122,"threads_started":1}
I20260812 02:41:09.525638 31646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:09.511453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:43365 (local address 127.3.127.131:37377)
0812 02:41:09.511799 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:09.511816 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:09.511862 (+ 46us) server_negotiation.cc:407] Connection header received
0812 02:41:09.512683 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:09.512691 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:09.513055 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:09.513378 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:09.514745 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:09.515769 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:09.517073 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:09.517617 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:09.517794 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:09.520565 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:09.520571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:09.520585 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:09.520619 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:09.522418 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:09.522969 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:09.522974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:09.522980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:09.523122 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:09.523476 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:09.523479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:09.523480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:09.524117 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:09.524255 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:09.525033 (+ 778us) server_negotiation.cc:299] Negotiation successful
0812 02:41:09.525407 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":73,"threads_started":1}
I20260812 02:41:09.526140 31649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:09.511279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:39947 (local address 127.3.127.129:55873)
0812 02:41:09.511881 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:09.511895 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:09.511998 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:09.513459 (+ 1461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:09.513465 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:09.514022 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:41:09.514551 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:09.514567 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:09.515932 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:09.515935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:09.516935 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:09.516946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:09.517749 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:09.517753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:09.517988 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:09.518609 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:41:09.518627 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:41:09.520477 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:41:09.522856 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:09.522859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:09.522861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:09.523054 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:09.523425 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:09.523428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:09.523430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:09.523511 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:09.524240 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:41:09.524246 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:41:09.524633 (+ 387us) client_negotiation.cc:769] Sending connection context
0812 02:41:09.524872 (+ 239us) client_negotiation.cc:240] Negotiation successful
0812 02:41:09.525393 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":570,"spinlock_wait_cycles":42368,"thread_start_us":69,"threads_started":1}
W20260812 02:41:09.528326 30998 leader_election.cc:341] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af2edfdfdc664199b20e4efdc6656c48 (127.3.127.131:37377): Not found: Tablet not found: a877ae3e8700493b89a336e69bf1fe3e
W20260812 02:41:09.528506 30990 leader_election.cc:341] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 019982c5360e4273a895747e6dec7c03 (127.3.127.130:39947): Not found: Tablet not found: a877ae3e8700493b89a336e69bf1fe3e
I20260812 02:41:09.528584 30990 leader_election.cc:302] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b21083b8daab4989b997fc4d9d660227; no voters: 019982c5360e4273a895747e6dec7c03, af2edfdfdc664199b20e4efdc6656c48
I20260812 02:41:09.528864 31645 raft_consensus.cc:2726] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 02:41:09.684633 31650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:09.684177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.131:51807)
0812 02:41:09.684432 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:09.684508 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
I20260812 02:41:10.327457 31651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:10.326973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.129:42377)
0812 02:41:10.327281 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:10.327353 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":85,"threads_started":1}
I20260812 02:41:10.468350 31652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:10.467862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.130:52477)
0812 02:41:10.468181 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:10.468246 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":97,"threads_started":1}
I20260812 02:41:10.685925 31654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:10.685451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.131:60795)
0812 02:41:10.685740 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:10.685803 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":85,"threads_started":1}
I20260812 02:41:11.328910 31655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:11.328364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.129:49369)
0812 02:41:11.328730 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:11.328805 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":86,"threads_started":1}
I20260812 02:41:11.469846 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:11.469354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.130:41607)
0812 02:41:11.469650 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:11.469718 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":76,"threads_started":1}
I20260812 02:41:11.687431 31659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:11.686884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.131:44503)
0812 02:41:11.687254 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:11.687328 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":84,"threads_started":1}
I20260812 02:41:11.759984 31660 raft_consensus.cc:482] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:41:11.760147 31660 raft_consensus.cc:504] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } }
I20260812 02:41:11.760677 31660 leader_election.cc:288] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af2edfdfdc664199b20e4efdc6656c48 (127.3.127.131:37377), 019982c5360e4273a895747e6dec7c03 (127.3.127.130:39947)
W20260812 02:41:11.761760 30990 leader_election.cc:341] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 019982c5360e4273a895747e6dec7c03 (127.3.127.130:39947): Not found: Tablet not found: a877ae3e8700493b89a336e69bf1fe3e
W20260812 02:41:11.761916 30998 leader_election.cc:341] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af2edfdfdc664199b20e4efdc6656c48 (127.3.127.131:37377): Not found: Tablet not found: a877ae3e8700493b89a336e69bf1fe3e
I20260812 02:41:11.762023 30998 leader_election.cc:302] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b21083b8daab4989b997fc4d9d660227; no voters: 019982c5360e4273a895747e6dec7c03, af2edfdfdc664199b20e4efdc6656c48
I20260812 02:41:11.762220 31660 raft_consensus.cc:2726] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 02:41:12.330386 31662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:12.329866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.129:42275)
0812 02:41:12.330207 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:12.330281 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":86,"threads_started":1}
I20260812 02:41:12.471210 31663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:12.470735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.130:51409)
0812 02:41:12.471028 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:12.471098 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":84,"threads_started":1}
I20260812 02:41:12.688884 31664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:12.688365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.131:34131)
0812 02:41:12.688709 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:12.688780 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":117,"threads_started":1}
I20260812 02:41:13.332136 31665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:13.331559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.129:46527)
0812 02:41:13.331957 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:13.332032 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":103,"threads_started":1}
I20260812 02:41:13.472669 31666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:13.472214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.130:58499)
0812 02:41:13.472496 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:13.472566 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":84,"threads_started":1}
I20260812 02:41:13.690250 31667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:13.689756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.131:57529)
0812 02:41:13.690070 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:13.690145 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":95,"threads_started":1}
I20260812 02:41:14.333967 31668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:14.333093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.129:39677)
0812 02:41:14.333763 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:14.333840 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":177,"threads_started":1}
I20260812 02:41:14.473945 31670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:14.473512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.130:44337)
0812 02:41:14.473781 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:14.473843 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":78,"threads_started":1}
I20260812 02:41:14.691567 31671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:14.691115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.131:59761)
0812 02:41:14.691402 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:14.691465 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":79,"threads_started":1}
I20260812 02:41:15.118129 31672 raft_consensus.cc:482] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:41:15.118300 31672 raft_consensus.cc:504] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } }
I20260812 02:41:15.118861 31672 leader_election.cc:288] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af2edfdfdc664199b20e4efdc6656c48 (127.3.127.131:37377), 019982c5360e4273a895747e6dec7c03 (127.3.127.130:39947)
W20260812 02:41:15.119755 30990 leader_election.cc:341] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 019982c5360e4273a895747e6dec7c03 (127.3.127.130:39947): Not found: Tablet not found: a877ae3e8700493b89a336e69bf1fe3e
W20260812 02:41:15.119882 30998 leader_election.cc:341] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af2edfdfdc664199b20e4efdc6656c48 (127.3.127.131:37377): Not found: Tablet not found: a877ae3e8700493b89a336e69bf1fe3e
I20260812 02:41:15.119946 30998 leader_election.cc:302] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b21083b8daab4989b997fc4d9d660227; no voters: 019982c5360e4273a895747e6dec7c03, af2edfdfdc664199b20e4efdc6656c48
I20260812 02:41:15.120088 31672 raft_consensus.cc:2726] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 02:41:15.335251 31673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:15.334773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.129:54791)
0812 02:41:15.335087 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:15.335151 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":80,"threads_started":1}
I20260812 02:41:15.475419 31674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:15.474844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.130:37801)
0812 02:41:15.475239 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:15.475317 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":91,"threads_started":1}
I20260812 02:41:15.692999 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:15.692459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.131:55223)
0812 02:41:15.692806 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:15.692876 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":85,"threads_started":1}
I20260812 02:41:16.336608 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:16.336084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.129:60233)
0812 02:41:16.336426 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:16.336497 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":107,"threads_started":1}
I20260812 02:41:16.476871 31681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:16.476378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.130:37167)
0812 02:41:16.476695 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:16.476767 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":100,"threads_started":1}
I20260812 02:41:16.694427 31683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:16.693913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.131:35141)
0812 02:41:16.694252 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:16.694321 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":103,"threads_started":1}
I20260812 02:41:17.337924 31684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:17.337443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.129:52265)
0812 02:41:17.337752 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:17.337819 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":83,"threads_started":1}
I20260812 02:41:17.478355 31686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:17.477855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.130:44629)
0812 02:41:17.478179 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:17.478250 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":88,"threads_started":1}
I20260812 02:41:17.695905 31687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:17.695342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.131:48051)
0812 02:41:17.695721 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:17.695794 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":91,"threads_started":1}
I20260812 02:41:18.339501 31688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:18.338971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.129:47575)
0812 02:41:18.339314 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:18.339391 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":82,"threads_started":1}
I20260812 02:41:18.479887 31689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:18.479351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.130:56175)
0812 02:41:18.479711 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:18.479782 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":127,"threads_started":1}
I20260812 02:41:18.697400 31691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:18.696896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.131:42469)
0812 02:41:18.697229 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:18.697296 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":120,"threads_started":1}
I20260812 02:41:18.939356 31692 raft_consensus.cc:482] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:41:18.939505 31692 raft_consensus.cc:504] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2edfdfdc664199b20e4efdc6656c48" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 37377 } } peers { permanent_uuid: "019982c5360e4273a895747e6dec7c03" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 39947 } } peers { permanent_uuid: "b21083b8daab4989b997fc4d9d660227" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 41547 } }
I20260812 02:41:18.939949 31692 leader_election.cc:288] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af2edfdfdc664199b20e4efdc6656c48 (127.3.127.131:37377), 019982c5360e4273a895747e6dec7c03 (127.3.127.130:39947)
W20260812 02:41:18.940842 30990 leader_election.cc:341] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 019982c5360e4273a895747e6dec7c03 (127.3.127.130:39947): Not found: Tablet not found: a877ae3e8700493b89a336e69bf1fe3e
W20260812 02:41:18.940968 30998 leader_election.cc:341] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af2edfdfdc664199b20e4efdc6656c48 (127.3.127.131:37377): Not found: Tablet not found: a877ae3e8700493b89a336e69bf1fe3e
I20260812 02:41:18.941033 30998 leader_election.cc:302] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b21083b8daab4989b997fc4d9d660227; no voters: 019982c5360e4273a895747e6dec7c03, af2edfdfdc664199b20e4efdc6656c48
I20260812 02:41:18.941217 31692 raft_consensus.cc:2726] T a877ae3e8700493b89a336e69bf1fe3e P b21083b8daab4989b997fc4d9d660227 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 02:41:19.340837 31693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:19.340367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.129:32915)
0812 02:41:19.340668 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:19.340732 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":104,"threads_started":1}
I20260812 02:41:19.481540 31694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:19.480951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.130:50109)
0812 02:41:19.481329 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:19.481401 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":102,"threads_started":1}
I20260812 02:41:19.698840 31697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:19.698330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.131:56271)
0812 02:41:19.698659 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:19.698731 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":97,"threads_started":1}
I20260812 02:41:20.342134 31699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:20.341692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.129:39257)
0812 02:41:20.341962 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:20.342021 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
I20260812 02:41:20.482913 31701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:20.482412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.130:40973)
0812 02:41:20.482734 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:20.482808 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":91,"threads_started":1}
I20260812 02:41:20.700129 31703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:20.699636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.131:58857)
0812 02:41:20.699935 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:20.700019 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":87,"threads_started":1}
I20260812 02:41:21.343626 31704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:21.343112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.129:51649)
0812 02:41:21.343437 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:21.343516 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":88,"threads_started":1}
I20260812 02:41:21.484336 31705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:21.483832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.130:49797)
0812 02:41:21.484158 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:21.484229 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":93,"threads_started":1}
I20260812 02:41:21.701695 31706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:21.701125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.131:60029)
0812 02:41:21.701481 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:21.701553 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":89,"threads_started":1}
I20260812 02:41:22.345115 31708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:22.344608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.129:45583)
0812 02:41:22.344927 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:22.345009 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":85,"threads_started":1}
I20260812 02:41:22.485659 31710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:22.485209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.130:33841)
0812 02:41:22.485479 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:22.485541 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260812 02:41:22.703107 31711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:22.702632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.131:57201)
0812 02:41:22.702927 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:22.702998 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":88,"threads_started":1}
I20260812 02:41:23.346836 31715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:23.346342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.129:43609)
0812 02:41:23.346664 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:23.346735 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":82,"threads_started":1}
I20260812 02:41:23.486964 31716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:23.486520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.130:53179)
0812 02:41:23.486803 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:23.486866 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":78,"threads_started":1}
I20260812 02:41:23.696164 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 30898
I20260812 02:41:23.706144 31717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:23.705651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:32981 (local address 127.3.127.131:57253)
0812 02:41:23.705968 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:23.706038 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.190:32981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":85,"threads_started":1}
I20260812 02:41:23.711637 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 31198
I20260812 02:41:23.729519 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 31422
2026-08-12T02:41:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59813 ms)
[----------] 30 tests from SecurityITest (259298 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:41:23 dist-test-slave-rmhg krb5kdc[31721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:41:23 dist-test-slave-rmhg krb5kdc[31721](info): set up 2 sockets
Aug 12 02:41:23 dist-test-slave-rmhg krb5kdc[31721](info): commencing operation
krb5kdc: starting...
W20260812 02:41:24.914714 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:41:25 dist-test-slave-rmhg krb5kdc[31721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:41:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:41:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:41:25.048491 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:42523
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:42523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:25.166615 31741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:25.166917 31741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:25.166957 31741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:25.171046 31741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:25.171118 31741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:25.171137 31741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:25.171152 31741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:25.171166 31741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:25.176620 31741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:42523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:42523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:25.177809 31741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:25.178853 31741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:25.187789 31758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:25.187858 31754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:25.187789 31753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:25.188036 31741 server_base.cc:1034] running on GCE node
I20260812 02:41:25.188674 31741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:25.189883 31741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:25.191058 31741 hybrid_clock.cc:648] HybridClock initialized: now 1786502485191032 us; error 41 us; skew 500 ppm
Aug 12 02:41:25 dist-test-slave-rmhg krb5kdc[31721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502485, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:25.193842 31741 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:41:25.194788 31741 webserver.cc:460] Webserver started at http://127.3.127.190:38001/ using document root <none> and password file <none>
I20260812 02:41:25.195335 31741 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:25.195397 31741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:25.195583 31741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:25.197749 31741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "989bf4762c914cff98d48ee90d41994e"
format_stamp: "Formatted at 2026-08-12 02:41:25 on dist-test-slave-rmhg"
server_key: "50bde5c7cb99d74411872902b5e1be58"
server_key_iv: "252ce1a1959b0d66304ee422d9b0ce0a"
server_key_version: "encryptionkey@0"
I20260812 02:41:25.198299 31741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "989bf4762c914cff98d48ee90d41994e"
format_stamp: "Formatted at 2026-08-12 02:41:25 on dist-test-slave-rmhg"
server_key: "50bde5c7cb99d74411872902b5e1be58"
server_key_iv: "252ce1a1959b0d66304ee422d9b0ce0a"
server_key_version: "encryptionkey@0"
I20260812 02:41:25.202068 31741 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:41:25.204531 31769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:25.205834 31741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 02:41:25.205948 31741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "989bf4762c914cff98d48ee90d41994e"
format_stamp: "Formatted at 2026-08-12 02:41:25 on dist-test-slave-rmhg"
server_key: "50bde5c7cb99d74411872902b5e1be58"
server_key_iv: "252ce1a1959b0d66304ee422d9b0ce0a"
server_key_version: "encryptionkey@0"
I20260812 02:41:25.206060 31741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:25.226307 31741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:25.229862 31741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:25.230065 31741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:25.237680 31741 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:42523
I20260812 02:41:25.237754 31851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:42523 every 8 connection(s)
I20260812 02:41:25.238741 31741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:25.241814 31852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:25.246558 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 31741
I20260812 02:41:25.246721 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:41:25.247032 3582 external_mini_cluster.cc:1428] Setting key 7a97cfede1b3fd6e3bad03289fcb9472
I20260812 02:41:25.248600 31852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e: Bootstrap starting.
I20260812 02:41:25.251235 31852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:25.252255 31852 log.cc:824] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:25.254468 31852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e: No bootstrap required, opened a new log
Aug 12 02:41:25 dist-test-slave-rmhg krb5kdc[31721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:41:25.258737 31852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "989bf4762c914cff98d48ee90d41994e" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 42523 } }
I20260812 02:41:25.259127 31852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:25.259227 31852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 989bf4762c914cff98d48ee90d41994e, State: Initialized, Role: FOLLOWER
I20260812 02:41:25.259853 31852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "989bf4762c914cff98d48ee90d41994e" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 42523 } }
I20260812 02:41:25.260000 31852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:25.260069 31852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:25.260185 31852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:25.261124 31852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "989bf4762c914cff98d48ee90d41994e" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 42523 } }
I20260812 02:41:25.261510 31852 leader_election.cc:302] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 989bf4762c914cff98d48ee90d41994e; no voters:
I20260812 02:41:25.261842 31852 leader_election.cc:288] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:41:25.261936 31860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:25.262148 31860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e [term 1 LEADER]: Becoming Leader. State: Replica: 989bf4762c914cff98d48ee90d41994e, State: Running, Role: LEADER
I20260812 02:41:25.262529 31860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "989bf4762c914cff98d48ee90d41994e" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 42523 } }
I20260812 02:41:25.263319 31852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:25.263758 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:25.248602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.3.127.190:42523)
0812 02:41:25.249181 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:25.249204 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:25.249254 (+ 50us) server_negotiation.cc:407] Connection header received
0812 02:41:25.250236 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:25.250264 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:25.250735 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:25.251156 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:25.252065 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:25.253731 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:25.254949 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:25.255671 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:25.255883 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:25.258401 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:25.258408 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:25.258427 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:25.258463 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:25.260584 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:25.261041 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:25.261048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:25.261055 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:25.261247 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:25.261525 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:25.261528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:25.261530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:25.262058 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:25.262333 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:25.262564 (+ 231us) server_negotiation.cc:299] Negotiation successful
0812 02:41:25.262915 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":147,"threads_started":1}
I20260812 02:41:25.263885 31860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 989bf4762c914cff98d48ee90d41994e. Latest consensus state: current_term: 1 leader_uuid: "989bf4762c914cff98d48ee90d41994e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "989bf4762c914cff98d48ee90d41994e" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 42523 } } }
I20260812 02:41:25.264139 31860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:25.264355 31860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "989bf4762c914cff98d48ee90d41994e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "989bf4762c914cff98d48ee90d41994e" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 42523 } } }
I20260812 02:41:25.264436 31860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:25.264771 31873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:25.268154 31873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:25.274649 31873 catalog_manager.cc:1347] Generated new cluster ID: 8e8dbe6124fd4b51a796923071bf796f
I20260812 02:41:25.274734 31873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:25.285099 31873 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:41:25.286479 31873 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:25.296576 31873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e: Generated new TSK 0
I20260812 02:41:25.297536 31873 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
I20260812 02:41:25.340058 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:42523
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:25.494912 31896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:25.495203 31896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:25.495247 31896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:25.499315 31896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:25.499388 31896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:25.499450 31896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:41:25.504914 31896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:42523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:25.506140 31896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:25.507202 31896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:25.517489 31911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:25.517482 31905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:25.517544 31896 server_base.cc:1034] running on GCE node
W20260812 02:41:25.517477 31908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:25.518074 31896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:25.518880 31896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:25.520072 31896 hybrid_clock.cc:648] HybridClock initialized: now 1786502485520057 us; error 28 us; skew 500 ppm
Aug 12 02:41:25 dist-test-slave-rmhg krb5kdc[31721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502485, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:25.523087 31896 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:41:25.524060 31896 webserver.cc:460] Webserver started at http://127.3.127.129:34753/ using document root <none> and password file <none>
I20260812 02:41:25.524660 31896 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:25.524727 31896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:25.524917 31896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:25.527233 31896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "5b0492599995458da0c5eeae672ed4a0"
format_stamp: "Formatted at 2026-08-12 02:41:25 on dist-test-slave-rmhg"
server_key: "ce48791883ba3397764c1e224f7005f4"
server_key_iv: "e112383603a6ef0892033cb2c89889b3"
server_key_version: "encryptionkey@0"
I20260812 02:41:25.527863 31896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "5b0492599995458da0c5eeae672ed4a0"
format_stamp: "Formatted at 2026-08-12 02:41:25 on dist-test-slave-rmhg"
server_key: "ce48791883ba3397764c1e224f7005f4"
server_key_iv: "e112383603a6ef0892033cb2c89889b3"
server_key_version: "encryptionkey@0"
I20260812 02:41:25.532183 31896 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:41:25.535028 31920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:25.536399 31896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:41:25.536510 31896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "5b0492599995458da0c5eeae672ed4a0"
format_stamp: "Formatted at 2026-08-12 02:41:25 on dist-test-slave-rmhg"
server_key: "ce48791883ba3397764c1e224f7005f4"
server_key_iv: "e112383603a6ef0892033cb2c89889b3"
server_key_version: "encryptionkey@0"
I20260812 02:41:25.536630 31896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:25.564929 31896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:25.568387 31896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:25.568588 31896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:25.569285 31896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:41:25.570200 31896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:41:25.570250 31896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:25.570294 31896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:41:25.570325 31896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:25.579174 31896 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:43035
I20260812 02:41:25.579222 32110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:43035 every 8 connection(s)
I20260812 02:41:25.580174 31896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
Aug 12 02:41:25 dist-test-slave-rmhg krb5kdc[31721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502485, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:41:25.589970 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 31896
I20260812 02:41:25.590101 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:41:25.590469 3582 external_mini_cluster.cc:1428] Setting key e4625332a99019bd5c663408655a2fde
I20260812 02:41:25.594579 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:25.582103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:49463 (local address 127.3.127.190:42523)
0812 02:41:25.582254 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:25.582257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:25.583142 (+ 885us) server_negotiation.cc:407] Connection header received
0812 02:41:25.584499 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:25.584503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:25.584564 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:25.584669 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:25.585966 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:25.586504 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:25.587663 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:25.587905 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:25.587998 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:25.590721 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:25.590725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:25.590727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:25.590756 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:25.592291 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:25.592877 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:25.592880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:25.592882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:25.592981 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:25.593354 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:25.593357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:25.593359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:25.593590 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:25.593695 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:25.594264 (+ 569us) server_negotiation.cc:299] Negotiation successful
0812 02:41:25.594410 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 02:41:25.595546 32114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:25.582504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:42523 (local address 127.3.127.129:49463)
0812 02:41:25.582983 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:25.583029 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:25.584264 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:25.584861 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:25.584868 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:25.585302 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:41:25.585818 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:25.585833 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:25.586630 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:25.586632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:25.587539 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:25.587549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:25.588058 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:25.588062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:25.588241 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:25.588781 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:41:25.588797 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:41:25.590516 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:41:25.592428 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:25.592433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:25.592448 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:25.592772 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:25.593069 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:25.593071 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:25.593072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:25.593271 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:25.593705 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:41:25.593711 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:41:25.593986 (+ 275us) client_negotiation.cc:769] Sending connection context
0812 02:41:25.594185 (+ 199us) client_negotiation.cc:240] Negotiation successful
0812 02:41:25.594607 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":123,"threads_started":1}
I20260812 02:41:25.596894 32111 heartbeater.cc:347] Connected to a master server at 127.3.127.190:42523
I20260812 02:41:25.597198 32111 heartbeater.cc:464] Registering TS with master...
I20260812 02:41:25.597759 32111 heartbeater.cc:511] Master 127.3.127.190:42523 requested a full tablet report, sending...
I20260812 02:41:25.599578 31803 ts_manager.cc:194] Registered new tserver with Master: 5b0492599995458da0c5eeae672ed4a0 (127.3.127.129:43035)
I20260812 02:41:25.601263 31803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:49463
WARNING: no policy specified for kudu/127.3.127.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
I20260812 02:41:25.622192 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:42523
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:25.741700 32117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:25.742002 32117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:25.742053 32117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:25.746282 32117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:25.746361 32117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:25.746423 32117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:41:25.752139 32117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:42523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:25.753463 32117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:25.754644 32117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:41:25.765334 32117 server_base.cc:1034] running on GCE node
W20260812 02:41:25.765256 32131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:25.765247 32126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:25.765247 32125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:25.766037 32117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:25.766808 32117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:25.767987 32117 hybrid_clock.cc:648] HybridClock initialized: now 1786502485767959 us; error 42 us; skew 500 ppm
Aug 12 02:41:25 dist-test-slave-rmhg krb5kdc[31721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502485, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:25.771209 32117 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:41:25.772244 32117 webserver.cc:460] Webserver started at http://127.3.127.130:34601/ using document root <none> and password file <none>
I20260812 02:41:25.772825 32117 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:25.772887 32117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:25.773067 32117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:25.775300 32117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "a57128661b944437ab6e6ce47161fa75"
format_stamp: "Formatted at 2026-08-12 02:41:25 on dist-test-slave-rmhg"
server_key: "690f9dc14686afe20bee78cc34478881"
server_key_iv: "b5d4a7b678e5452a71d445a6f0ff814a"
server_key_version: "encryptionkey@0"
I20260812 02:41:25.775854 32117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "a57128661b944437ab6e6ce47161fa75"
format_stamp: "Formatted at 2026-08-12 02:41:25 on dist-test-slave-rmhg"
server_key: "690f9dc14686afe20bee78cc34478881"
server_key_iv: "b5d4a7b678e5452a71d445a6f0ff814a"
server_key_version: "encryptionkey@0"
I20260812 02:41:25.779901 32117 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 02:41:25.782809 32140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:25.784214 32117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:41:25.784359 32117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "a57128661b944437ab6e6ce47161fa75"
format_stamp: "Formatted at 2026-08-12 02:41:25 on dist-test-slave-rmhg"
server_key: "690f9dc14686afe20bee78cc34478881"
server_key_iv: "b5d4a7b678e5452a71d445a6f0ff814a"
server_key_version: "encryptionkey@0"
I20260812 02:41:25.784497 32117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:25.815249 32117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:25.818823 32117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:25.819024 32117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:25.819700 32117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:41:25.820636 32117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:41:25.820686 32117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:25.820734 32117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:41:25.820763 32117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:25.829905 32117 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:42155
I20260812 02:41:25.829932 32329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:42155 every 8 connection(s)
I20260812 02:41:25.830870 32117 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
Aug 12 02:41:25 dist-test-slave-rmhg krb5kdc[31721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502485, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:41:25.840989 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 32117
I20260812 02:41:25.841144 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:41:25.841534 3582 external_mini_cluster.cc:1428] Setting key 4325b7eb6cac85c821c452e61e6da2ab
I20260812 02:41:25.845547 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:25.832791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:60245 (local address 127.3.127.190:42523)
0812 02:41:25.832975 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:25.832980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:25.834053 (+ 1073us) server_negotiation.cc:407] Connection header received
0812 02:41:25.835343 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:25.835346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:25.835400 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:25.835480 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:25.836737 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:25.837329 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:25.838496 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:25.838767 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:25.838854 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:25.841536 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:25.841539 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:25.841541 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:25.841571 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:25.843220 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:25.843775 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:25.843779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:25.843781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:25.843880 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:25.844281 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:25.844284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:25.844286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:25.844536 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:25.844643 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:25.845248 (+ 605us) server_negotiation.cc:299] Negotiation successful
0812 02:41:25.845396 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 02:41:25.846822 32332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:25.833344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:42523 (local address 127.3.127.130:60245)
0812 02:41:25.833884 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:25.833933 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:25.835109 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:25.835669 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:25.835677 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:25.836060 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:41:25.836577 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:25.836593 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:25.837422 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:25.837425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:25.838373 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:25.838383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:25.838903 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:25.838906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:25.839059 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:25.839627 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:41:25.839645 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:41:25.841302 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:41:25.843340 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:25.843346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:25.843362 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:25.843661 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:25.844007 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:25.844010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:25.844012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:25.844186 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:25.844658 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:41:25.844665 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:41:25.844959 (+ 294us) client_negotiation.cc:769] Sending connection context
0812 02:41:25.845143 (+ 184us) client_negotiation.cc:240] Negotiation successful
0812 02:41:25.845661 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":126,"threads_started":1}
I20260812 02:41:25.848237 32330 heartbeater.cc:347] Connected to a master server at 127.3.127.190:42523
I20260812 02:41:25.848511 32330 heartbeater.cc:464] Registering TS with master...
I20260812 02:41:25.849061 32330 heartbeater.cc:511] Master 127.3.127.190:42523 requested a full tablet report, sending...
I20260812 02:41:25.850309 31803 ts_manager.cc:194] Registered new tserver with Master: a57128661b944437ab6e6ce47161fa75 (127.3.127.130:42155)
I20260812 02:41:25.851141 31803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:60245
WARNING: no policy specified for kudu/127.3.127.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
I20260812 02:41:25.874058 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:42523
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:25.992247 32337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:25.992549 32337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:25.992594 32337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:25.996744 32337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:25.996820 32337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:25.996881 32337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:41:26.002547 32337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:42523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:26.003683 32337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:26.004765 32337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:26.015030 32344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:26.014976 32345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:26.014971 32348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:26.015383 32337 server_base.cc:1034] running on GCE node
I20260812 02:41:26.015858 32337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:26.016551 32337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:26.017730 32337 hybrid_clock.cc:648] HybridClock initialized: now 1786502486017704 us; error 38 us; skew 500 ppm
Aug 12 02:41:26 dist-test-slave-rmhg krb5kdc[31721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502486, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:26.020650 32337 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:41:26.021692 32337 webserver.cc:460] Webserver started at http://127.3.127.131:36767/ using document root <none> and password file <none>
I20260812 02:41:26.022279 32337 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:26.022349 32337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:26.022562 32337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:26.024724 32337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "9840c2ff7e2c488da44c011788946015"
format_stamp: "Formatted at 2026-08-12 02:41:26 on dist-test-slave-rmhg"
server_key: "a9e3ca68ae75d9c3c538ebc7b76cc307"
server_key_iv: "8c99362561ecd8e40e02b73a5c8f6066"
server_key_version: "encryptionkey@0"
I20260812 02:41:26.025348 32337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "9840c2ff7e2c488da44c011788946015"
format_stamp: "Formatted at 2026-08-12 02:41:26 on dist-test-slave-rmhg"
server_key: "a9e3ca68ae75d9c3c538ebc7b76cc307"
server_key_iv: "8c99362561ecd8e40e02b73a5c8f6066"
server_key_version: "encryptionkey@0"
I20260812 02:41:26.029143 32337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:41:26.031807 32362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:26.033221 32337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:41:26.033351 32337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "9840c2ff7e2c488da44c011788946015"
format_stamp: "Formatted at 2026-08-12 02:41:26 on dist-test-slave-rmhg"
server_key: "a9e3ca68ae75d9c3c538ebc7b76cc307"
server_key_iv: "8c99362561ecd8e40e02b73a5c8f6066"
server_key_version: "encryptionkey@0"
I20260812 02:41:26.033461 32337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:26.050545 32337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:26.054311 32337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:26.054514 32337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:26.055164 32337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:41:26.056056 32337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:41:26.056113 32337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:26.056175 32337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:41:26.056207 32337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:26.065454 32337 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:42675
I20260812 02:41:26.065503 32545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:42675 every 8 connection(s)
I20260812 02:41:26.066433 32337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:41:26.071935 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 32337
I20260812 02:41:26.072073 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:41:26.072408 3582 external_mini_cluster.cc:1428] Setting key 83c9e042845ff3e9ef12c1ed9d46e92d
Aug 12 02:41:26 dist-test-slave-rmhg krb5kdc[31721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502486, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:41:26.080636 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:26.068296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:51369 (local address 127.3.127.190:42523)
0812 02:41:26.068434 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:26.068438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:26.069402 (+ 964us) server_negotiation.cc:407] Connection header received
0812 02:41:26.070702 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:26.070706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:26.070768 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:26.070847 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:26.072135 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.072876 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.074242 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.074462 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.074555 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:26.077092 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:26.077095 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:26.077097 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:26.077125 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:26.078437 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:26.078992 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:26.078995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:26.078996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:26.079086 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:26.079446 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:26.079450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:26.079452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:26.079659 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:26.079751 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:26.080315 (+ 564us) server_negotiation.cc:299] Negotiation successful
0812 02:41:26.080463 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 02:41:26.081637 32549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:26.068715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:42523 (local address 127.3.127.131:51369)
0812 02:41:26.069248 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:26.069294 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:26.070480 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:26.071043 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:26.071051 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:26.071429 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:41:26.071950 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:26.071964 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.073024 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.073027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:26.074114 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:26.074125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.074609 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.074614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:26.074805 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:26.075358 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:41:26.075377 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:41:26.076901 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:41:26.078572 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:26.078579 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:26.078597 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:26.078875 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:26.079178 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:26.079181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:26.079183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:26.079349 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:26.079768 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:41:26.079776 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:41:26.080055 (+ 279us) client_negotiation.cc:769] Sending connection context
0812 02:41:26.080238 (+ 183us) client_negotiation.cc:240] Negotiation successful
0812 02:41:26.080609 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":117,"threads_started":1}
I20260812 02:41:26.082850 32546 heartbeater.cc:347] Connected to a master server at 127.3.127.190:42523
I20260812 02:41:26.083101 32546 heartbeater.cc:464] Registering TS with master...
I20260812 02:41:26.083624 32546 heartbeater.cc:511] Master 127.3.127.190:42523 requested a full tablet report, sending...
I20260812 02:41:26.084846 31803 ts_manager.cc:194] Registered new tserver with Master: 9840c2ff7e2c488da44c011788946015 (127.3.127.131:42675)
I20260812 02:41:26.085619 31803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:51369
I20260812 02:41:26.086452 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:41:26.096266 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:26.088914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.3.127.190:42523)
0812 02:41:26.089092 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:26.089096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:26.089247 (+ 151us) server_negotiation.cc:407] Connection header received
0812 02:41:26.089373 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:26.089376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:26.089419 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:26.089487 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:26.090169 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.090680 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.091543 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.091778 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.091866 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:26.093257 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:26.093260 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:26.093262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:26.093286 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:26.094469 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:26.094899 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:26.094903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:26.094904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:26.095000 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:26.095392 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:26.095394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:26.095395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:26.095642 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:26.095760 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:26.095944 (+ 184us) server_negotiation.cc:299] Negotiation successful
0812 02:41:26.096075 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 02:41:26.099689 31803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:41:26.102844 31803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:41:26.118741 32575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:26.112151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:42155 (local address 127.0.0.1:44776)
0812 02:41:26.113057 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:26.113115 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:26.113325 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:26.113905 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:26.113912 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:26.113936 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:26.114340 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:26.114349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.115871 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.115875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:26.117419 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:26.117432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.118302 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.118305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:26.118390 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:26.118413 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 02:41:26.118452 (+ 39us) client_negotiation.cc:240] Negotiation successful
0812 02:41:26.118577 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":212,"thread_start_us":113,"threads_started":1}
I20260812 02:41:26.118930 32574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:26.112192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38452 (local address 127.3.127.129:43035)
0812 02:41:26.113631 (+ 1439us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:26.113640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:26.113665 (+ 25us) server_negotiation.cc:407] Connection header received
0812 02:41:26.113737 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:26.113740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:26.113910 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:26.114350 (+ 440us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:26.114501 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.115728 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.117555 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.118321 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.118425 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:26.118591 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:26.118675 (+ 84us) server_negotiation.cc:299] Negotiation successful
0812 02:41:26.118781 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1326,"thread_start_us":89,"threads_started":1}
I20260812 02:41:26.119155 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:26.112851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44776 (local address 127.3.127.130:42155)
0812 02:41:26.113331 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:26.113345 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:26.113372 (+ 27us) server_negotiation.cc:407] Connection header received
0812 02:41:26.113456 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:26.113462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:26.113724 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:26.113908 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:26.114476 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.115719 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.117575 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.118171 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.118566 (+ 395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:26.118709 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:26.118761 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 02:41:26.118965 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":96,"threads_started":1}
I20260812 02:41:26.119313 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:26.112014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:43035 (local address 127.0.0.1:38452)
0812 02:41:26.113229 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:26.113244 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:26.113324 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:26.114072 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:26.114075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:26.114091 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:26.114193 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:26.114200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.115871 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.115875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:26.117426 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:26.117432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.118443 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.118446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:26.118547 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:26.118572 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 02:41:26.118643 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 02:41:26.118718 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":135,"thread_start_us":96,"threads_started":1}
I20260812 02:41:26.119473 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:26.112806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55208 (local address 127.3.127.131:42675)
0812 02:41:26.113632 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:26.113642 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:26.114020 (+ 378us) server_negotiation.cc:407] Connection header received
0812 02:41:26.114121 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:26.114126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:26.114269 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:26.114407 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:26.114683 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.115442 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.117225 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.117978 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.118107 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:26.118279 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:26.119212 (+ 933us) server_negotiation.cc:299] Negotiation successful
0812 02:41:26.119333 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":177,"threads_started":1}
I20260812 02:41:26.119781 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:26.112646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:42675 (local address 127.0.0.1:55208)
0812 02:41:26.113827 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:26.113841 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:26.113922 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:26.114407 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:26.114410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:26.114427 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:26.114521 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:26.114527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.115587 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.115590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:26.117093 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:26.117104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.118126 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.118130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:26.118256 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:26.118310 (+ 54us) client_negotiation.cc:769] Sending connection context
0812 02:41:26.118417 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 02:41:26.119662 (+ 1245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"mutex_wait_us":308,"thread_start_us":122,"threads_started":1}
I20260812 02:41:26.122756 32438 tablet_service.cc:1467] Processing CreateTablet for tablet 71a876da82954a34baf652d07d79ac52 (DEFAULT_TABLE table=test-table [id=c856cb93107642899cec1aef7fb6da05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:41:26.122762 32223 tablet_service.cc:1467] Processing CreateTablet for tablet 71a876da82954a34baf652d07d79ac52 (DEFAULT_TABLE table=test-table [id=c856cb93107642899cec1aef7fb6da05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:41:26.123164 31998 tablet_service.cc:1467] Processing CreateTablet for tablet 71a876da82954a34baf652d07d79ac52 (DEFAULT_TABLE table=test-table [id=c856cb93107642899cec1aef7fb6da05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:41:26.123888 32223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a876da82954a34baf652d07d79ac52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:26.123906 32438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a876da82954a34baf652d07d79ac52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:26.124081 31998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a876da82954a34baf652d07d79ac52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:26.130281 32582 tablet_bootstrap.cc:492] T 71a876da82954a34baf652d07d79ac52 P 9840c2ff7e2c488da44c011788946015: Bootstrap starting.
I20260812 02:41:26.131944 32584 tablet_bootstrap.cc:492] T 71a876da82954a34baf652d07d79ac52 P 5b0492599995458da0c5eeae672ed4a0: Bootstrap starting.
I20260812 02:41:26.133111 32582 tablet_bootstrap.cc:654] T 71a876da82954a34baf652d07d79ac52 P 9840c2ff7e2c488da44c011788946015: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:26.133530 32585 tablet_bootstrap.cc:492] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75: Bootstrap starting.
I20260812 02:41:26.134282 32584 tablet_bootstrap.cc:654] T 71a876da82954a34baf652d07d79ac52 P 5b0492599995458da0c5eeae672ed4a0: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:26.134690 32582 log.cc:824] T 71a876da82954a34baf652d07d79ac52 P 9840c2ff7e2c488da44c011788946015: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:26.135162 32584 log.cc:824] T 71a876da82954a34baf652d07d79ac52 P 5b0492599995458da0c5eeae672ed4a0: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:26.135869 32585 tablet_bootstrap.cc:654] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:26.136718 32585 log.cc:824] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:26.136870 32582 tablet_bootstrap.cc:492] T 71a876da82954a34baf652d07d79ac52 P 9840c2ff7e2c488da44c011788946015: No bootstrap required, opened a new log
I20260812 02:41:26.137053 32582 ts_tablet_manager.cc:1397] T 71a876da82954a34baf652d07d79ac52 P 9840c2ff7e2c488da44c011788946015: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 02:41:26.137347 32584 tablet_bootstrap.cc:492] T 71a876da82954a34baf652d07d79ac52 P 5b0492599995458da0c5eeae672ed4a0: No bootstrap required, opened a new log
I20260812 02:41:26.137569 32584 ts_tablet_manager.cc:1397] T 71a876da82954a34baf652d07d79ac52 P 5b0492599995458da0c5eeae672ed4a0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 02:41:26.138914 32585 tablet_bootstrap.cc:492] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75: No bootstrap required, opened a new log
I20260812 02:41:26.139119 32585 ts_tablet_manager.cc:1397] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 02:41:26.140399 32582 raft_consensus.cc:348] T 71a876da82954a34baf652d07d79ac52 P 9840c2ff7e2c488da44c011788946015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0492599995458da0c5eeae672ed4a0" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43035 } } peers { permanent_uuid: "a57128661b944437ab6e6ce47161fa75" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 42155 } } peers { permanent_uuid: "9840c2ff7e2c488da44c011788946015" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42675 } }
I20260812 02:41:26.140666 32582 raft_consensus.cc:374] T 71a876da82954a34baf652d07d79ac52 P 9840c2ff7e2c488da44c011788946015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:26.140782 32582 raft_consensus.cc:729] T 71a876da82954a34baf652d07d79ac52 P 9840c2ff7e2c488da44c011788946015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9840c2ff7e2c488da44c011788946015, State: Initialized, Role: FOLLOWER
I20260812 02:41:26.141281 32582 consensus_queue.cc:260] T 71a876da82954a34baf652d07d79ac52 P 9840c2ff7e2c488da44c011788946015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0492599995458da0c5eeae672ed4a0" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43035 } } peers { permanent_uuid: "a57128661b944437ab6e6ce47161fa75" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 42155 } } peers { permanent_uuid: "9840c2ff7e2c488da44c011788946015" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42675 } }
I20260812 02:41:26.142135 32546 heartbeater.cc:503] Master 127.3.127.190:42523 was elected leader, sending a full tablet report...
I20260812 02:41:26.142158 32584 raft_consensus.cc:348] T 71a876da82954a34baf652d07d79ac52 P 5b0492599995458da0c5eeae672ed4a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0492599995458da0c5eeae672ed4a0" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43035 } } peers { permanent_uuid: "a57128661b944437ab6e6ce47161fa75" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 42155 } } peers { permanent_uuid: "9840c2ff7e2c488da44c011788946015" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42675 } }
I20260812 02:41:26.142429 32584 raft_consensus.cc:374] T 71a876da82954a34baf652d07d79ac52 P 5b0492599995458da0c5eeae672ed4a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:26.142473 32584 raft_consensus.cc:729] T 71a876da82954a34baf652d07d79ac52 P 5b0492599995458da0c5eeae672ed4a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0492599995458da0c5eeae672ed4a0, State: Initialized, Role: FOLLOWER
I20260812 02:41:26.142932 32584 consensus_queue.cc:260] T 71a876da82954a34baf652d07d79ac52 P 5b0492599995458da0c5eeae672ed4a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0492599995458da0c5eeae672ed4a0" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43035 } } peers { permanent_uuid: "a57128661b944437ab6e6ce47161fa75" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 42155 } } peers { permanent_uuid: "9840c2ff7e2c488da44c011788946015" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42675 } }
I20260812 02:41:26.143018 32585 raft_consensus.cc:348] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0492599995458da0c5eeae672ed4a0" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43035 } } peers { permanent_uuid: "a57128661b944437ab6e6ce47161fa75" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 42155 } } peers { permanent_uuid: "9840c2ff7e2c488da44c011788946015" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42675 } }
I20260812 02:41:26.143296 32585 raft_consensus.cc:374] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:26.143348 32585 raft_consensus.cc:729] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57128661b944437ab6e6ce47161fa75, State: Initialized, Role: FOLLOWER
I20260812 02:41:26.143630 32582 ts_tablet_manager.cc:1428] T 71a876da82954a34baf652d07d79ac52 P 9840c2ff7e2c488da44c011788946015: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 02:41:26.143798 32585 consensus_queue.cc:260] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0492599995458da0c5eeae672ed4a0" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43035 } } peers { permanent_uuid: "a57128661b944437ab6e6ce47161fa75" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 42155 } } peers { permanent_uuid: "9840c2ff7e2c488da44c011788946015" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42675 } }
I20260812 02:41:26.143970 32584 ts_tablet_manager.cc:1428] T 71a876da82954a34baf652d07d79ac52 P 5b0492599995458da0c5eeae672ed4a0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 02:41:26.144304 32111 heartbeater.cc:503] Master 127.3.127.190:42523 was elected leader, sending a full tablet report...
I20260812 02:41:26.144745 32330 heartbeater.cc:503] Master 127.3.127.190:42523 was elected leader, sending a full tablet report...
I20260812 02:41:26.144783 32585 ts_tablet_manager.cc:1428] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 02:41:26.208426 32591 raft_consensus.cc:482] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:41:26.208675 32591 raft_consensus.cc:504] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0492599995458da0c5eeae672ed4a0" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43035 } } peers { permanent_uuid: "a57128661b944437ab6e6ce47161fa75" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 42155 } } peers { permanent_uuid: "9840c2ff7e2c488da44c011788946015" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42675 } }
I20260812 02:41:26.210163 32591 leader_election.cc:288] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b0492599995458da0c5eeae672ed4a0 (127.3.127.129:43035), 9840c2ff7e2c488da44c011788946015 (127.3.127.131:42675)
I20260812 02:41:26.214407 32574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:26.210572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:49821 (local address 127.3.127.129:43035)
0812 02:41:26.210838 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:26.210843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:26.210861 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:41:26.211041 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:26.211045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:26.211100 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:26.211180 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:26.211506 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.212281 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.213400 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.213848 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.213968 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:26.214030 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:26.214208 (+ 178us) server_negotiation.cc:299] Negotiation successful
0812 02:41:26.214294 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260812 02:41:26.214407 32332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:26.210398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:43035 (local address 127.3.127.130:49821)
0812 02:41:26.210769 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:26.210785 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:26.210885 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:26.211205 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:26.211209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:26.211236 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:26.211369 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:26.211376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.212374 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.212376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:26.213266 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:26.213277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.213969 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.213973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:26.214083 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:26.214104 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 02:41:26.214177 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 02:41:26.214260 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281}
I20260812 02:41:26.214594 32596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:26.210412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:42675 (local address 127.3.127.130:54395)
0812 02:41:26.211181 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:26.211197 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:26.211340 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:26.211602 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:26.211610 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:26.211627 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:26.211727 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:26.211732 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.212598 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.212601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:26.213472 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:26.213483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.214163 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.214167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:26.214283 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:26.214302 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 02:41:26.214369 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 02:41:26.214444 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":124,"thread_start_us":191,"threads_started":1}
I20260812 02:41:26.214577 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:26.210622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:54395 (local address 127.3.127.131:42675)
0812 02:41:26.210797 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:26.210801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:26.211271 (+ 470us) server_negotiation.cc:407] Connection header received
0812 02:41:26.211453 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:26.211456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:26.211510 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:26.211582 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:26.211823 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.212482 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.213598 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.214029 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.214144 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:26.214199 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:26.214397 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 02:41:26.214467 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260812 02:41:26.215207 32027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a876da82954a34baf652d07d79ac52" candidate_uuid: "a57128661b944437ab6e6ce47161fa75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0492599995458da0c5eeae672ed4a0" is_pre_election: true
I20260812 02:41:26.215217 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a876da82954a34baf652d07d79ac52" candidate_uuid: "a57128661b944437ab6e6ce47161fa75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9840c2ff7e2c488da44c011788946015" is_pre_election: true
I20260812 02:41:26.215526 32474 raft_consensus.cc:2454] T 71a876da82954a34baf652d07d79ac52 P 9840c2ff7e2c488da44c011788946015 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a57128661b944437ab6e6ce47161fa75 in term 0.
I20260812 02:41:26.215531 32027 raft_consensus.cc:2454] T 71a876da82954a34baf652d07d79ac52 P 5b0492599995458da0c5eeae672ed4a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a57128661b944437ab6e6ce47161fa75 in term 0.
I20260812 02:41:26.216061 32146 leader_election.cc:302] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b0492599995458da0c5eeae672ed4a0, a57128661b944437ab6e6ce47161fa75; no voters:
I20260812 02:41:26.216365 32591 raft_consensus.cc:2781] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:41:26.216470 32591 raft_consensus.cc:482] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:41:26.216512 32591 raft_consensus.cc:3037] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:26.217685 32591 raft_consensus.cc:504] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0492599995458da0c5eeae672ed4a0" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43035 } } peers { permanent_uuid: "a57128661b944437ab6e6ce47161fa75" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 42155 } } peers { permanent_uuid: "9840c2ff7e2c488da44c011788946015" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42675 } }
I20260812 02:41:26.218240 32591 leader_election.cc:288] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 [CANDIDATE]: Term 1 election: Requested vote from peers 5b0492599995458da0c5eeae672ed4a0 (127.3.127.129:43035), 9840c2ff7e2c488da44c011788946015 (127.3.127.131:42675)
I20260812 02:41:26.218628 32027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a876da82954a34baf652d07d79ac52" candidate_uuid: "a57128661b944437ab6e6ce47161fa75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0492599995458da0c5eeae672ed4a0"
I20260812 02:41:26.218680 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a876da82954a34baf652d07d79ac52" candidate_uuid: "a57128661b944437ab6e6ce47161fa75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9840c2ff7e2c488da44c011788946015"
I20260812 02:41:26.218781 32027 raft_consensus.cc:3037] T 71a876da82954a34baf652d07d79ac52 P 5b0492599995458da0c5eeae672ed4a0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:26.218819 32474 raft_consensus.cc:3037] T 71a876da82954a34baf652d07d79ac52 P 9840c2ff7e2c488da44c011788946015 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:26.219962 32027 raft_consensus.cc:2454] T 71a876da82954a34baf652d07d79ac52 P 5b0492599995458da0c5eeae672ed4a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a57128661b944437ab6e6ce47161fa75 in term 1.
I20260812 02:41:26.219964 32474 raft_consensus.cc:2454] T 71a876da82954a34baf652d07d79ac52 P 9840c2ff7e2c488da44c011788946015 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a57128661b944437ab6e6ce47161fa75 in term 1.
I20260812 02:41:26.220357 32146 leader_election.cc:302] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5b0492599995458da0c5eeae672ed4a0, 9840c2ff7e2c488da44c011788946015, a57128661b944437ab6e6ce47161fa75; no voters:
I20260812 02:41:26.220556 32591 raft_consensus.cc:2781] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:26.220868 32591 raft_consensus.cc:686] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 [term 1 LEADER]: Becoming Leader. State: Replica: a57128661b944437ab6e6ce47161fa75, State: Running, Role: LEADER
I20260812 02:41:26.221311 32591 consensus_queue.cc:237] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0492599995458da0c5eeae672ed4a0" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43035 } } peers { permanent_uuid: "a57128661b944437ab6e6ce47161fa75" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 42155 } } peers { permanent_uuid: "9840c2ff7e2c488da44c011788946015" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42675 } }
I20260812 02:41:26.225070 31801 catalog_manager.cc:5495] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a57128661b944437ab6e6ce47161fa75 (127.3.127.130). New cstate: current_term: 1 leader_uuid: "a57128661b944437ab6e6ce47161fa75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0492599995458da0c5eeae672ed4a0" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a57128661b944437ab6e6ce47161fa75" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 42155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9840c2ff7e2c488da44c011788946015" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42675 } health_report { overall_health: UNKNOWN } } }
I20260812 02:41:26.244699 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:26.240364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.3.127.130:42155)
0812 02:41:26.240547 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:26.240551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:26.240565 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:41:26.240651 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:26.240653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:26.240716 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:26.240809 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:41:26.241135 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.241743 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.243040 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.243276 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.243364 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:26.243771 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:41:26.244234 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:41:26.244354 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:26.244406 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 02:41:26.244493 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 02:41:26.251881 32474 raft_consensus.cc:1261] T 71a876da82954a34baf652d07d79ac52 P 9840c2ff7e2c488da44c011788946015 [term 1 FOLLOWER]: Refusing update from remote peer a57128661b944437ab6e6ce47161fa75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:41:26.251881 32027 raft_consensus.cc:1261] T 71a876da82954a34baf652d07d79ac52 P 5b0492599995458da0c5eeae672ed4a0 [term 1 FOLLOWER]: Refusing update from remote peer a57128661b944437ab6e6ce47161fa75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:41:26.252765 32598 consensus_queue.cc:1035] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b0492599995458da0c5eeae672ed4a0" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:41:26.252985 32591 consensus_queue.cc:1035] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9840c2ff7e2c488da44c011788946015" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:41:26.262306 32604 mvcc.cc:205] Tried to move back new op lower bound from 7317514183678537728 to 7317514183565479936. Current Snapshot: MvccSnapshot[applied={T|T < 7317514183678537728}]
I20260812 02:41:26.263068 32603 mvcc.cc:205] Tried to move back new op lower bound from 7317514183678537728 to 7317514183565479936. Current Snapshot: MvccSnapshot[applied={T|T < 7317514183678537728}]
I20260812 02:41:26.273785 31801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:41:26.273973 31801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:41:26.276866 31801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 989bf4762c914cff98d48ee90d41994e: Sending DeleteTablet for 3 replicas of tablet 71a876da82954a34baf652d07d79ac52
I20260812 02:41:26.277850 32438 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a876da82954a34baf652d07d79ac52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:41:26 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:55208
I20260812 02:41:26.277849 32223 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a876da82954a34baf652d07d79ac52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:41:26 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:44776
I20260812 02:41:26.277882 31998 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a876da82954a34baf652d07d79ac52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:41:26 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:38452
I20260812 02:41:26.278493 32627 tablet_replica.cc:330] T 71a876da82954a34baf652d07d79ac52 P 9840c2ff7e2c488da44c011788946015: stopping tablet replica
I20260812 02:41:26.278702 32628 tablet_replica.cc:330] T 71a876da82954a34baf652d07d79ac52 P 5b0492599995458da0c5eeae672ed4a0: stopping tablet replica
I20260812 02:41:26.279071 32627 raft_consensus.cc:2229] T 71a876da82954a34baf652d07d79ac52 P 9840c2ff7e2c488da44c011788946015 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:41:26.279104 32628 raft_consensus.cc:2229] T 71a876da82954a34baf652d07d79ac52 P 5b0492599995458da0c5eeae672ed4a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:41:26.279474 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 31896
I20260812 02:41:26.279486 32627 raft_consensus.cc:2258] T 71a876da82954a34baf652d07d79ac52 P 9840c2ff7e2c488da44c011788946015 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:41:26.296315 32627 ts_tablet_manager.cc:1905] T 71a876da82954a34baf652d07d79ac52 P 9840c2ff7e2c488da44c011788946015: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:41:26.296891 32626 tablet_replica.cc:330] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75: stopping tablet replica
I20260812 02:41:26.297436 32626 raft_consensus.cc:2229] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:41:26.297998 32626 raft_consensus.cc:2258] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:41:26.299738 32627 ts_tablet_manager.cc:1918] T 71a876da82954a34baf652d07d79ac52 P 9840c2ff7e2c488da44c011788946015: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:41:26.299866 32627 log.cc:1196] T 71a876da82954a34baf652d07d79ac52 P 9840c2ff7e2c488da44c011788946015: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/71a876da82954a34baf652d07d79ac52
I20260812 02:41:26.299979 32626 ts_tablet_manager.cc:1905] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:41:26.300275 32627 ts_tablet_manager.cc:1939] T 71a876da82954a34baf652d07d79ac52 P 9840c2ff7e2c488da44c011788946015: Deleting consensus metadata
I20260812 02:41:26.301805 31777 catalog_manager.cc:4841] TS 9840c2ff7e2c488da44c011788946015 (127.3.127.131:42675): tablet 71a876da82954a34baf652d07d79ac52 (table test-table [id=c856cb93107642899cec1aef7fb6da05]) successfully deleted
I20260812 02:41:26.303184 32626 ts_tablet_manager.cc:1918] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260812 02:41:26.303218 31776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.127.129:43035 (error 108)
I20260812 02:41:26.303315 32626 log.cc:1196] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/71a876da82954a34baf652d07d79ac52
I20260812 02:41:26.303653 32626 ts_tablet_manager.cc:1939] T 71a876da82954a34baf652d07d79ac52 P a57128661b944437ab6e6ce47161fa75: Deleting consensus metadata
I20260812 02:41:26.304415 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:26.304116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:43035 (local address 127.0.0.1:38466)
0812 02:41:26.304247 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:26.304321 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.129:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260812 02:41:26.304569 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 32117
W20260812 02:41:26.324153 31776 catalog_manager.cc:4584] TS 5b0492599995458da0c5eeae672ed4a0 (127.3.127.129:43035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71a876da82954a34baf652d07d79ac52: Network error: Client connection negotiation failed: client connection to 127.3.127.129:43035: connect: Connection refused (error 111)
I20260812 02:41:26.336666 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 32337
I20260812 02:41:26.340291 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:26.336916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:42155 (local address 127.0.0.1:44792)
0812 02:41:26.340087 (+ 3171us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:26.340175 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.130:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3025}
I20260812 02:41:26.346405 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 31741
2026-08-12T02:41:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2622 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T02:41:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:41:26Z Disabled control of system clock
I20260812 02:41:26.390543 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:35665
--webserver_interface=127.3.127.190
--webserver_port=0
--builtin_ntp_servers=127.3.127.148:39293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:35665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 02:41:26.507315 32633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:26.507614 32633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:26.507650 32633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:26.511804 32633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:26.511884 32633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:26.511899 32633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:26.511915 32633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:26.511929 32633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:26.516980 32633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:35665
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:35665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:26.518149 32633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:26.519543 32633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:26.528625 32638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:26.528614 32639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:26.528614 32642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:26.528678 32633 server_base.cc:1034] running on GCE node
I20260812 02:41:26.529601 32633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:26.530897 32633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:26.532079 32633 hybrid_clock.cc:648] HybridClock initialized: now 1786502486532067 us; error 40 us; skew 500 ppm
I20260812 02:41:26.534186 32633 webserver.cc:460] Webserver started at http://127.3.127.190:46521/ using document root <none> and password file <none>
I20260812 02:41:26.534803 32633 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:26.534878 32633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:26.535096 32633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:26.537431 32633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "6f98e56c0ffc4fee8bf75f5f8ef1adb3"
format_stamp: "Formatted at 2026-08-12 02:41:26 on dist-test-slave-rmhg"
server_key: "f2a3dd1897053494e2c1c40f3d62bb9c"
server_key_iv: "d85ed0ef4914f27bb08f4df8d6405018"
server_key_version: "encryptionkey@0"
I20260812 02:41:26.538079 32633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "6f98e56c0ffc4fee8bf75f5f8ef1adb3"
format_stamp: "Formatted at 2026-08-12 02:41:26 on dist-test-slave-rmhg"
server_key: "f2a3dd1897053494e2c1c40f3d62bb9c"
server_key_iv: "d85ed0ef4914f27bb08f4df8d6405018"
server_key_version: "encryptionkey@0"
I20260812 02:41:26.542192 32633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:41:26.545014 32654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:26.546425 32633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:41:26.546564 32633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "6f98e56c0ffc4fee8bf75f5f8ef1adb3"
format_stamp: "Formatted at 2026-08-12 02:41:26 on dist-test-slave-rmhg"
server_key: "f2a3dd1897053494e2c1c40f3d62bb9c"
server_key_iv: "d85ed0ef4914f27bb08f4df8d6405018"
server_key_version: "encryptionkey@0"
I20260812 02:41:26.546679 32633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:26.591064 32633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:26.591850 32633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:26.592054 32633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:26.599560 32633 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:35665
I20260812 02:41:26.599561 32741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:35665 every 8 connection(s)
I20260812 02:41:26.600629 32633 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:26.604080 32742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:26.608105 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 32633
I20260812 02:41:26.608261 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:41:26.608632 3582 external_mini_cluster.cc:1428] Setting key d889f732bd2f1ebec8ebee25174891b6
I20260812 02:41:26.610684 32742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3: Bootstrap starting.
I20260812 02:41:26.613250 32742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:26.614053 32742 log.cc:824] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:26.616422 32742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3: No bootstrap required, opened a new log
I20260812 02:41:26.618389 32746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:26.610286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51700 (local address 127.3.127.190:35665)
0812 02:41:26.610975 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:26.610993 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:26.611031 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:41:26.611726 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:26.611745 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:26.612181 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:26.612474 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:26.612842 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.613927 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.615001 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.615501 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.615644 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:26.616486 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:26.616493 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:26.616512 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:26.616947 (+ 435us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:26.617008 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:26.617030 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:26.617113 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:26.617328 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 02:41:26.617610 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":140,"threads_started":1}
I20260812 02:41:26.620513 32742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f98e56c0ffc4fee8bf75f5f8ef1adb3" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 35665 } }
I20260812 02:41:26.620783 32742 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:26.620848 32742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f98e56c0ffc4fee8bf75f5f8ef1adb3, State: Initialized, Role: FOLLOWER
I20260812 02:41:26.621332 32742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f98e56c0ffc4fee8bf75f5f8ef1adb3" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 35665 } }
I20260812 02:41:26.621486 32742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:26.621570 32742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:26.621680 32742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:26.622632 32742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f98e56c0ffc4fee8bf75f5f8ef1adb3" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 35665 } }
I20260812 02:41:26.622972 32742 leader_election.cc:302] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f98e56c0ffc4fee8bf75f5f8ef1adb3; no voters:
I20260812 02:41:26.623284 32742 leader_election.cc:288] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:41:26.623409 32753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:26.623610 32753 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3 [term 1 LEADER]: Becoming Leader. State: Replica: 6f98e56c0ffc4fee8bf75f5f8ef1adb3, State: Running, Role: LEADER
I20260812 02:41:26.623925 32753 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f98e56c0ffc4fee8bf75f5f8ef1adb3" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 35665 } }
I20260812 02:41:26.624691 32742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:26.625753 32755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f98e56c0ffc4fee8bf75f5f8ef1adb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f98e56c0ffc4fee8bf75f5f8ef1adb3" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 35665 } } }
I20260812 02:41:26.625886 32755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:26.626173 32756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f98e56c0ffc4fee8bf75f5f8ef1adb3. Latest consensus state: current_term: 1 leader_uuid: "6f98e56c0ffc4fee8bf75f5f8ef1adb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f98e56c0ffc4fee8bf75f5f8ef1adb3" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 35665 } } }
I20260812 02:41:26.626281 32756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:26.626319 32767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:26.629457 32767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:26.635025 32767 catalog_manager.cc:1347] Generated new cluster ID: 6225366b3f4b4c1da648e20a73c87c8c
I20260812 02:41:26.635097 32767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:26.657084 32767 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:41:26.658118 32767 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:26.668292 32767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3: Generated new TSK 0
I20260812 02:41:26.669313 32767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:41:26.674955 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:35665
--builtin_ntp_servers=127.3.127.148:39293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 02:41:26.797405 313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:26.797745 313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:26.797811 313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:26.802129 313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:26.802199 313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:26.802281 313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:41:26.807394 313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:35665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:26.808624 313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:26.810154 313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:26.820537 322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:26.820511 324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:26.820917 318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:26.821064 313 server_base.cc:1034] running on GCE node
I20260812 02:41:26.821624 313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:26.822391 313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:26.823614 313 hybrid_clock.cc:648] HybridClock initialized: now 1786502486823561 us; error 69 us; skew 500 ppm
I20260812 02:41:26.837455 313 webserver.cc:460] Webserver started at http://127.3.127.129:45785/ using document root <none> and password file <none>
I20260812 02:41:26.838153 313 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:26.838234 313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:26.838445 313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:26.840757 313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "3cbb0b1c115d45bdbc43f54bfdb680bd"
format_stamp: "Formatted at 2026-08-12 02:41:26 on dist-test-slave-rmhg"
server_key: "7f2877bd0593d559e3bd112d13c6d8e2"
server_key_iv: "57ed6f8b0eb50d7b37e6ebb90f95362c"
server_key_version: "encryptionkey@0"
I20260812 02:41:26.841359 313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "3cbb0b1c115d45bdbc43f54bfdb680bd"
format_stamp: "Formatted at 2026-08-12 02:41:26 on dist-test-slave-rmhg"
server_key: "7f2877bd0593d559e3bd112d13c6d8e2"
server_key_iv: "57ed6f8b0eb50d7b37e6ebb90f95362c"
server_key_version: "encryptionkey@0"
I20260812 02:41:26.845353 313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:41:26.848019 336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:26.849442 313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:41:26.849568 313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "3cbb0b1c115d45bdbc43f54bfdb680bd"
format_stamp: "Formatted at 2026-08-12 02:41:26 on dist-test-slave-rmhg"
server_key: "7f2877bd0593d559e3bd112d13c6d8e2"
server_key_iv: "57ed6f8b0eb50d7b37e6ebb90f95362c"
server_key_version: "encryptionkey@0"
I20260812 02:41:26.849682 313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:26.877992 313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:26.878767 313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:26.878958 313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:26.879580 313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:41:26.880599 313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:41:26.880661 313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:26.880719 313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:41:26.880746 313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:26.889876 313 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:39387
I20260812 02:41:26.889922 526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:39387 every 8 connection(s)
I20260812 02:41:26.890836 313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:41:26.893529 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 313
I20260812 02:41:26.893668 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:41:26.894004 3582 external_mini_cluster.cc:1428] Setting key 55025d972fb9ff73c9973b0739ecf2c8
I20260812 02:41:26.896361 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:35665
--builtin_ntp_servers=127.3.127.148:39293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 02:41:26.901234 32746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:26.892708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:35889 (local address 127.3.127.190:35665)
0812 02:41:26.892880 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:26.892884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:26.894230 (+ 1346us) server_negotiation.cc:407] Connection header received
0812 02:41:26.895908 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:26.895911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:26.895961 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:26.896025 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:26.897262 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.897790 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.899113 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.899442 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.899557 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:26.900700 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:26.900705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:26.900707 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:26.900723 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:26.900750 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:26.900763 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:26.900846 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:26.900978 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 02:41:26.901050 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 02:41:26.902210 533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:26.893210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:35665 (local address 127.3.127.129:35889)
0812 02:41:26.894051 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:26.894114 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:26.895675 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:26.896238 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:26.896248 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:26.896365 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:41:26.897094 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:26.897115 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.897922 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.897925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:26.898977 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:26.898987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:26.899588 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:26.899591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:26.899776 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:26.900383 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:41:26.900385 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:41:26.900429 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:41:26.900536 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:41:26.900849 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:41:26.900854 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:41:26.900858 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 02:41:26.900951 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 02:41:26.901291 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":113,"threads_started":1}
I20260812 02:41:26.903484 530 heartbeater.cc:347] Connected to a master server at 127.3.127.190:35665
I20260812 02:41:26.903735 530 heartbeater.cc:464] Registering TS with master...
I20260812 02:41:26.904285 530 heartbeater.cc:511] Master 127.3.127.190:35665 requested a full tablet report, sending...
I20260812 02:41:26.905812 32679 ts_manager.cc:194] Registered new tserver with Master: 3cbb0b1c115d45bdbc43f54bfdb680bd (127.3.127.129:39387)
I20260812 02:41:26.906965 32679 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.127.129:35889
W20260812 02:41:27.014122 534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:27.014408 534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:27.014451 534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:27.018573 534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:27.018646 534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:27.018705 534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:41:27.023766 534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:35665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:27.025014 534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:27.026587 534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:41:27.037002 534 server_base.cc:1034] running on GCE node
W20260812 02:41:27.036937 541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:27.036937 547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:27.036972 540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:27.037688 534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:27.038385 534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:27.039544 534 hybrid_clock.cc:648] HybridClock initialized: now 1786502487039500 us; error 68 us; skew 500 ppm
I20260812 02:41:27.041538 534 webserver.cc:460] Webserver started at http://127.3.127.130:37783/ using document root <none> and password file <none>
I20260812 02:41:27.042135 534 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:27.042201 534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:27.042414 534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:27.044679 534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "d16f7d3207b94b6ab4b580c34a9f0762"
format_stamp: "Formatted at 2026-08-12 02:41:27 on dist-test-slave-rmhg"
server_key: "0f2dcebbbb9733ddb5b87bbe1132e326"
server_key_iv: "783b30b5a3be0d63184ae00dc85c0c28"
server_key_version: "encryptionkey@0"
I20260812 02:41:27.045295 534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "d16f7d3207b94b6ab4b580c34a9f0762"
format_stamp: "Formatted at 2026-08-12 02:41:27 on dist-test-slave-rmhg"
server_key: "0f2dcebbbb9733ddb5b87bbe1132e326"
server_key_iv: "783b30b5a3be0d63184ae00dc85c0c28"
server_key_version: "encryptionkey@0"
I20260812 02:41:27.049288 534 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:41:27.052028 556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:27.053476 534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:41:27.053604 534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "d16f7d3207b94b6ab4b580c34a9f0762"
format_stamp: "Formatted at 2026-08-12 02:41:27 on dist-test-slave-rmhg"
server_key: "0f2dcebbbb9733ddb5b87bbe1132e326"
server_key_iv: "783b30b5a3be0d63184ae00dc85c0c28"
server_key_version: "encryptionkey@0"
I20260812 02:41:27.053728 534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:27.081295 534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:27.082110 534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:27.082307 534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:27.082998 534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:41:27.084105 534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:41:27.084157 534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:27.084208 534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:41:27.084237 534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:27.093398 534 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:44729
I20260812 02:41:27.093422 761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:44729 every 8 connection(s)
I20260812 02:41:27.094421 534 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:41:27.094656 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 534
I20260812 02:41:27.094779 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:41:27.095122 3582 external_mini_cluster.cc:1428] Setting key 2507e49191bd19f79f9251943b18c90c
I20260812 02:41:27.097551 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:35665
--builtin_ntp_servers=127.3.127.148:39293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 02:41:27.105813 32746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:27.096512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:49807 (local address 127.3.127.190:35665)
0812 02:41:27.096632 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:27.096637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:27.098176 (+ 1539us) server_negotiation.cc:407] Connection header received
0812 02:41:27.099932 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:27.099936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:27.100005 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:27.100100 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:27.101255 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.101792 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.103585 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.103822 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.103909 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:27.105322 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:27.105326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:27.105328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:27.105343 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:27.105364 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:27.105374 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:27.105433 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:27.105619 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 02:41:27.105684 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 02:41:27.107194 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:27.097134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:35665 (local address 127.3.127.130:49807)
0812 02:41:27.097996 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:27.098053 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:27.099692 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:27.100336 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:27.100344 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:27.100439 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:41:27.101045 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:27.101068 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.101937 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.101941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:27.103448 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:27.103463 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.103998 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.104003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:27.104240 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:27.104935 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:41:27.104937 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:41:27.105006 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:41:27.105107 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:41:27.105456 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:41:27.105466 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:41:27.105472 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 02:41:27.105617 (+ 145us) client_negotiation.cc:240] Negotiation successful
0812 02:41:27.105976 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":143,"threads_started":1}
I20260812 02:41:27.108579 762 heartbeater.cc:347] Connected to a master server at 127.3.127.190:35665
I20260812 02:41:27.108892 762 heartbeater.cc:464] Registering TS with master...
I20260812 02:41:27.109697 762 heartbeater.cc:511] Master 127.3.127.190:35665 requested a full tablet report, sending...
I20260812 02:41:27.111106 32679 ts_manager.cc:194] Registered new tserver with Master: d16f7d3207b94b6ab4b580c34a9f0762 (127.3.127.130:44729)
I20260812 02:41:27.111709 32679 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.127.130:49807
W20260812 02:41:27.221879 766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:27.222203 766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:27.222265 766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:27.226557 766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:27.226627 766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:27.226713 766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:41:27.231802 766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:35665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:27.233248 766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:27.234720 766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:27.245343 776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:27.245345 775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:27.245483 766 server_base.cc:1034] running on GCE node
W20260812 02:41:27.245343 779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:27.246147 766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:27.246882 766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:27.248062 766 hybrid_clock.cc:648] HybridClock initialized: now 1786502487248034 us; error 41 us; skew 500 ppm
I20260812 02:41:27.250238 766 webserver.cc:460] Webserver started at http://127.3.127.131:45627/ using document root <none> and password file <none>
I20260812 02:41:27.250861 766 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:27.250937 766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:27.251186 766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:27.253669 766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "a1ed7aca49de49478adbba0e83185ff3"
format_stamp: "Formatted at 2026-08-12 02:41:27 on dist-test-slave-rmhg"
server_key: "f6e7a8ff6fcb23eb255184dcb92fa33b"
server_key_iv: "98feec010bd6c11389bac72bac98a983"
server_key_version: "encryptionkey@0"
I20260812 02:41:27.254294 766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "a1ed7aca49de49478adbba0e83185ff3"
format_stamp: "Formatted at 2026-08-12 02:41:27 on dist-test-slave-rmhg"
server_key: "f6e7a8ff6fcb23eb255184dcb92fa33b"
server_key_iv: "98feec010bd6c11389bac72bac98a983"
server_key_version: "encryptionkey@0"
I20260812 02:41:27.258733 766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:41:27.261452 790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:27.262825 766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 02:41:27.262969 766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "a1ed7aca49de49478adbba0e83185ff3"
format_stamp: "Formatted at 2026-08-12 02:41:27 on dist-test-slave-rmhg"
server_key: "f6e7a8ff6fcb23eb255184dcb92fa33b"
server_key_iv: "98feec010bd6c11389bac72bac98a983"
server_key_version: "encryptionkey@0"
I20260812 02:41:27.263090 766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:27.274525 766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:27.275317 766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:27.275509 766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:27.276144 766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:41:27.277201 766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:41:27.277253 766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:27.277311 766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:41:27.277335 766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:27.286269 766 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:45445
I20260812 02:41:27.286324 976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:45445 every 8 connection(s)
I20260812 02:41:27.287257 766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:41:27.296314 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 766
I20260812 02:41:27.296451 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:41:27.296789 3582 external_mini_cluster.cc:1428] Setting key dccd82d545e109c10f7baef693058911
I20260812 02:41:27.297138 32746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:27.289290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:56057 (local address 127.3.127.190:35665)
0812 02:41:27.289440 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:27.289444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:27.290622 (+ 1178us) server_negotiation.cc:407] Connection header received
0812 02:41:27.292024 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:27.292027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:27.292081 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:27.292169 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:27.293099 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.293718 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.294931 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.295183 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.295271 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:27.296573 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:27.296576 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:27.296578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:27.296594 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:27.296617 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:27.296626 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:27.296687 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:27.296902 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 02:41:27.296974 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 02:41:27.298231 983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:27.289746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:35665 (local address 127.3.127.131:56057)
0812 02:41:27.290453 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:27.290497 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:27.291804 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:27.292322 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:27.292329 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:27.292418 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:41:27.292975 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:27.292989 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.293847 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.293849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:27.294800 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:27.294811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.295354 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.295357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:27.295527 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:27.296208 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:41:27.296211 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:41:27.296275 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:41:27.296372 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:41:27.296717 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:41:27.296728 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:41:27.296735 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 02:41:27.296885 (+ 150us) client_negotiation.cc:240] Negotiation successful
0812 02:41:27.297271 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":145,"threads_started":1}
I20260812 02:41:27.299450 977 heartbeater.cc:347] Connected to a master server at 127.3.127.190:35665
I20260812 02:41:27.299712 977 heartbeater.cc:464] Registering TS with master...
I20260812 02:41:27.300277 977 heartbeater.cc:511] Master 127.3.127.190:35665 requested a full tablet report, sending...
I20260812 02:41:27.301429 32678 ts_manager.cc:194] Registered new tserver with Master: a1ed7aca49de49478adbba0e83185ff3 (127.3.127.131:45445)
I20260812 02:41:27.302028 32678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.127.131:56057
I20260812 02:41:27.311338 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:41:27.317268 32746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:27.313483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51714 (local address 127.3.127.190:35665)
0812 02:41:27.313682 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:27.313686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:27.313796 (+ 110us) server_negotiation.cc:407] Connection header received
0812 02:41:27.313959 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:27.313962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:27.314014 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:27.314091 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:27.314429 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.314976 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.315841 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.316049 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.316121 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:27.316850 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:27.316853 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:27.316855 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:27.316868 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:27.316888 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:27.316896 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:27.316949 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:27.317013 (+ 64us) server_negotiation.cc:299] Negotiation successful
0812 02:41:27.317069 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 02:41:27.320431 32678 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:41:27.323581 32678 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:41:27.339553 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:27.332851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:45445 (local address 127.0.0.1:44046)
0812 02:41:27.333364 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:27.333424 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:27.334304 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:27.335090 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:27.335093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:27.335123 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:27.335266 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:27.335273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.336611 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.336615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:27.337835 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:27.337848 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.338973 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.338976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:27.339091 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:27.339140 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 02:41:27.339242 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 02:41:27.339374 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":83,"spinlock_wait_cycles":3840,"thread_start_us":130,"threads_started":1}
I20260812 02:41:27.339812 1006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:27.333804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56320 (local address 127.3.127.130:44729)
0812 02:41:27.334248 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:27.334260 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:27.334288 (+ 28us) server_negotiation.cc:407] Connection header received
0812 02:41:27.334604 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:27.334609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:27.334769 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:27.334911 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:27.335229 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.336498 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.338146 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.338922 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.339013 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:27.339213 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:27.339551 (+ 338us) server_negotiation.cc:299] Negotiation successful
0812 02:41:27.339664 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":144,"threads_started":1}
I20260812 02:41:27.340286 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:27.333284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:44729 (local address 127.0.0.1:56320)
0812 02:41:27.333957 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:27.333970 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:27.334460 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:27.334912 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:27.334918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:27.334941 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:27.335089 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:27.335096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.336644 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.336647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:27.338024 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:27.338032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.339284 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.339288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:27.339376 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:27.339399 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 02:41:27.340061 (+ 662us) client_negotiation.cc:240] Negotiation successful
0812 02:41:27.340150 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":1,"spinlock_wait_cycles":898944,"thread_start_us":102,"threads_started":1}
I20260812 02:41:27.340512 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:27.333716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:39387 (local address 127.0.0.1:43636)
0812 02:41:27.334209 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:27.334221 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:27.334304 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:27.335090 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:27.335093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:27.335125 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:27.335269 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:27.335274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.338022 (+ 2748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.338026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:27.338787 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:27.338797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.340066 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.340069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:27.340185 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:27.340208 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 02:41:27.340274 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 02:41:27.340348 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":23,"spinlock_wait_cycles":5120,"thread_start_us":87,"threads_started":1}
I20260812 02:41:27.339589 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:27.332981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44046 (local address 127.3.127.131:45445)
0812 02:41:27.333766 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:27.333781 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:27.333807 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:41:27.334526 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:27.334533 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:27.334756 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:27.334917 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:27.335390 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.336513 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.337979 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.338848 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.338978 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:27.339213 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:27.339292 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 02:41:27.339406 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":210,"threads_started":1}
I20260812 02:41:27.341066 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:27.334153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43636 (local address 127.3.127.129:39387)
0812 02:41:27.334470 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:27.334481 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:27.334503 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:41:27.334581 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:27.334585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:27.334756 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:27.334917 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:27.336667 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.337879 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.338922 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.339704 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.339805 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:27.339998 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:27.340787 (+ 789us) server_negotiation.cc:299] Negotiation successful
0812 02:41:27.340904 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":114,"threads_started":1}
I20260812 02:41:27.343240 638 tablet_service.cc:1467] Processing CreateTablet for tablet 4ef6d110bc634a6eb870c8b5a25e4976 (DEFAULT_TABLE table=test-table [id=70254f8ac4ac49caac26a8ecf85607a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:41:27.343552 873 tablet_service.cc:1467] Processing CreateTablet for tablet 4ef6d110bc634a6eb870c8b5a25e4976 (DEFAULT_TABLE table=test-table [id=70254f8ac4ac49caac26a8ecf85607a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:41:27.343559 414 tablet_service.cc:1467] Processing CreateTablet for tablet 4ef6d110bc634a6eb870c8b5a25e4976 (DEFAULT_TABLE table=test-table [id=70254f8ac4ac49caac26a8ecf85607a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:41:27.344481 638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef6d110bc634a6eb870c8b5a25e4976. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:27.344722 414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef6d110bc634a6eb870c8b5a25e4976. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:27.345005 873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef6d110bc634a6eb870c8b5a25e4976. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:27.351982 1008 tablet_bootstrap.cc:492] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762: Bootstrap starting.
I20260812 02:41:27.352953 1010 tablet_bootstrap.cc:492] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd: Bootstrap starting.
I20260812 02:41:27.353554 1011 tablet_bootstrap.cc:492] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3: Bootstrap starting.
I20260812 02:41:27.355029 1010 tablet_bootstrap.cc:654] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:27.355057 1008 tablet_bootstrap.cc:654] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:27.355671 1011 tablet_bootstrap.cc:654] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:27.355981 1010 log.cc:824] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:27.356226 1008 log.cc:824] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:27.356454 1011 log.cc:824] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:27.358508 1008 tablet_bootstrap.cc:492] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762: No bootstrap required, opened a new log
I20260812 02:41:27.358533 1011 tablet_bootstrap.cc:492] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3: No bootstrap required, opened a new log
I20260812 02:41:27.358533 1010 tablet_bootstrap.cc:492] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd: No bootstrap required, opened a new log
I20260812 02:41:27.358739 1010 ts_tablet_manager.cc:1397] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 02:41:27.358739 1011 ts_tablet_manager.cc:1397] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260812 02:41:27.358740 1008 ts_tablet_manager.cc:1397] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 02:41:27.362234 1008 raft_consensus.cc:348] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ed7aca49de49478adbba0e83185ff3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 45445 } } peers { permanent_uuid: "3cbb0b1c115d45bdbc43f54bfdb680bd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 39387 } } peers { permanent_uuid: "d16f7d3207b94b6ab4b580c34a9f0762" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 44729 } }
I20260812 02:41:27.362514 1008 raft_consensus.cc:374] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:27.362583 1008 raft_consensus.cc:729] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d16f7d3207b94b6ab4b580c34a9f0762, State: Initialized, Role: FOLLOWER
I20260812 02:41:27.363106 1008 consensus_queue.cc:260] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ed7aca49de49478adbba0e83185ff3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 45445 } } peers { permanent_uuid: "3cbb0b1c115d45bdbc43f54bfdb680bd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 39387 } } peers { permanent_uuid: "d16f7d3207b94b6ab4b580c34a9f0762" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 44729 } }
I20260812 02:41:27.363152 1011 raft_consensus.cc:348] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ed7aca49de49478adbba0e83185ff3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 45445 } } peers { permanent_uuid: "3cbb0b1c115d45bdbc43f54bfdb680bd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 39387 } } peers { permanent_uuid: "d16f7d3207b94b6ab4b580c34a9f0762" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 44729 } }
I20260812 02:41:27.363176 1010 raft_consensus.cc:348] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ed7aca49de49478adbba0e83185ff3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 45445 } } peers { permanent_uuid: "3cbb0b1c115d45bdbc43f54bfdb680bd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 39387 } } peers { permanent_uuid: "d16f7d3207b94b6ab4b580c34a9f0762" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 44729 } }
I20260812 02:41:27.363437 1011 raft_consensus.cc:374] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:27.363437 1010 raft_consensus.cc:374] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:27.363487 1010 raft_consensus.cc:729] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cbb0b1c115d45bdbc43f54bfdb680bd, State: Initialized, Role: FOLLOWER
I20260812 02:41:27.363487 1011 raft_consensus.cc:729] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1ed7aca49de49478adbba0e83185ff3, State: Initialized, Role: FOLLOWER
I20260812 02:41:27.363958 1010 consensus_queue.cc:260] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ed7aca49de49478adbba0e83185ff3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 45445 } } peers { permanent_uuid: "3cbb0b1c115d45bdbc43f54bfdb680bd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 39387 } } peers { permanent_uuid: "d16f7d3207b94b6ab4b580c34a9f0762" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 44729 } }
I20260812 02:41:27.363998 1011 consensus_queue.cc:260] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ed7aca49de49478adbba0e83185ff3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 45445 } } peers { permanent_uuid: "3cbb0b1c115d45bdbc43f54bfdb680bd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 39387 } } peers { permanent_uuid: "d16f7d3207b94b6ab4b580c34a9f0762" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 44729 } }
I20260812 02:41:27.364349 1008 ts_tablet_manager.cc:1428] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260812 02:41:27.364799 762 heartbeater.cc:503] Master 127.3.127.190:35665 was elected leader, sending a full tablet report...
I20260812 02:41:27.365043 1010 ts_tablet_manager.cc:1428] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 02:41:27.365043 1011 ts_tablet_manager.cc:1428] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 02:41:27.365350 530 heartbeater.cc:503] Master 127.3.127.190:35665 was elected leader, sending a full tablet report...
I20260812 02:41:27.365358 977 heartbeater.cc:503] Master 127.3.127.190:35665 was elected leader, sending a full tablet report...
W20260812 02:41:27.392112 531 tablet.cc:2369] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:41:27.538524 981 tablet.cc:2369] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:41:27.595746 764 tablet.cc:2369] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:41:27.662976 1021 raft_consensus.cc:482] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:41:27.663241 1021 raft_consensus.cc:504] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ed7aca49de49478adbba0e83185ff3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 45445 } } peers { permanent_uuid: "3cbb0b1c115d45bdbc43f54bfdb680bd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 39387 } } peers { permanent_uuid: "d16f7d3207b94b6ab4b580c34a9f0762" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 44729 } }
I20260812 02:41:27.664484 1021 leader_election.cc:288] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1ed7aca49de49478adbba0e83185ff3 (127.3.127.131:45445), d16f7d3207b94b6ab4b580c34a9f0762 (127.3.127.130:44729)
I20260812 02:41:27.668573 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:27.664884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:57477 (local address 127.3.127.131:45445)
0812 02:41:27.665018 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:27.665022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:27.665315 (+ 293us) server_negotiation.cc:407] Connection header received
0812 02:41:27.665453 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:27.665458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:27.665514 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:27.665572 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:27.665883 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.666445 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.667627 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.668041 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.668176 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:27.668221 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:27.668391 (+ 170us) server_negotiation.cc:299] Negotiation successful
0812 02:41:27.668463 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 02:41:27.668592 1024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:27.664781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:45445 (local address 127.3.127.129:57477)
0812 02:41:27.665208 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:27.665221 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:27.665339 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:27.665600 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:27.665610 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:27.665626 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:27.665772 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:27.665778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.666553 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.666557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:27.667509 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:27.667519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.668162 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.668166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:27.668264 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:27.668288 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 02:41:27.668358 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 02:41:27.668431 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":6,"thread_start_us":99,"threads_started":1}
I20260812 02:41:27.668726 1006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:27.664884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:38439 (local address 127.3.127.130:44729)
0812 02:41:27.665018 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:27.665022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:27.665385 (+ 363us) server_negotiation.cc:407] Connection header received
0812 02:41:27.665493 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:27.665495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:27.665541 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:27.665614 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:27.665909 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.666422 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.667620 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.668064 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.668186 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:27.668236 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:27.668507 (+ 271us) server_negotiation.cc:299] Negotiation successful
0812 02:41:27.668584 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 02:41:27.669119 1023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:27.664781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:44729 (local address 127.3.127.129:38439)
0812 02:41:27.665262 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:27.665276 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:27.665404 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:27.665636 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:27.665668 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:27.665688 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:27.665788 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:27.665801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.666518 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.666521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:27.667502 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:27.667513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.668315 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.668317 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:27.668387 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:27.668405 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 02:41:27.668470 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 02:41:27.668538 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":95,"thread_start_us":87,"threads_started":1}
I20260812 02:41:27.669454 903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6d110bc634a6eb870c8b5a25e4976" candidate_uuid: "3cbb0b1c115d45bdbc43f54bfdb680bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1ed7aca49de49478adbba0e83185ff3" is_pre_election: true
I20260812 02:41:27.669749 670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6d110bc634a6eb870c8b5a25e4976" candidate_uuid: "3cbb0b1c115d45bdbc43f54bfdb680bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d16f7d3207b94b6ab4b580c34a9f0762" is_pre_election: true
I20260812 02:41:27.669798 903 raft_consensus.cc:2454] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cbb0b1c115d45bdbc43f54bfdb680bd in term 0.
I20260812 02:41:27.670012 670 raft_consensus.cc:2454] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cbb0b1c115d45bdbc43f54bfdb680bd in term 0.
I20260812 02:41:27.670300 341 leader_election.cc:302] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cbb0b1c115d45bdbc43f54bfdb680bd, a1ed7aca49de49478adbba0e83185ff3; no voters:
I20260812 02:41:27.670610 1021 raft_consensus.cc:2781] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:41:27.670704 1021 raft_consensus.cc:482] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:41:27.670760 1021 raft_consensus.cc:3037] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:27.671818 1021 raft_consensus.cc:504] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ed7aca49de49478adbba0e83185ff3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 45445 } } peers { permanent_uuid: "3cbb0b1c115d45bdbc43f54bfdb680bd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 39387 } } peers { permanent_uuid: "d16f7d3207b94b6ab4b580c34a9f0762" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 44729 } }
I20260812 02:41:27.672165 1021 leader_election.cc:288] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd [CANDIDATE]: Term 1 election: Requested vote from peers a1ed7aca49de49478adbba0e83185ff3 (127.3.127.131:45445), d16f7d3207b94b6ab4b580c34a9f0762 (127.3.127.130:44729)
I20260812 02:41:27.672490 670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6d110bc634a6eb870c8b5a25e4976" candidate_uuid: "3cbb0b1c115d45bdbc43f54bfdb680bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d16f7d3207b94b6ab4b580c34a9f0762"
I20260812 02:41:27.672515 903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6d110bc634a6eb870c8b5a25e4976" candidate_uuid: "3cbb0b1c115d45bdbc43f54bfdb680bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1ed7aca49de49478adbba0e83185ff3"
I20260812 02:41:27.672614 670 raft_consensus.cc:3037] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:27.672636 903 raft_consensus.cc:3037] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:27.673739 670 raft_consensus.cc:2454] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cbb0b1c115d45bdbc43f54bfdb680bd in term 1.
I20260812 02:41:27.673817 903 raft_consensus.cc:2454] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cbb0b1c115d45bdbc43f54bfdb680bd in term 1.
I20260812 02:41:27.674088 337 leader_election.cc:302] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cbb0b1c115d45bdbc43f54bfdb680bd, d16f7d3207b94b6ab4b580c34a9f0762; no voters:
I20260812 02:41:27.674288 1021 raft_consensus.cc:2781] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:27.674476 1021 raft_consensus.cc:686] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd [term 1 LEADER]: Becoming Leader. State: Replica: 3cbb0b1c115d45bdbc43f54bfdb680bd, State: Running, Role: LEADER
I20260812 02:41:27.674844 1021 consensus_queue.cc:237] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ed7aca49de49478adbba0e83185ff3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 45445 } } peers { permanent_uuid: "3cbb0b1c115d45bdbc43f54bfdb680bd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 39387 } } peers { permanent_uuid: "d16f7d3207b94b6ab4b580c34a9f0762" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 44729 } }
I20260812 02:41:27.678026 32677 catalog_manager.cc:5495] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cbb0b1c115d45bdbc43f54bfdb680bd (127.3.127.129). New cstate: current_term: 1 leader_uuid: "3cbb0b1c115d45bdbc43f54bfdb680bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ed7aca49de49478adbba0e83185ff3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cbb0b1c115d45bdbc43f54bfdb680bd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 39387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d16f7d3207b94b6ab4b580c34a9f0762" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 44729 } health_report { overall_health: UNKNOWN } } }
I20260812 02:41:27.706393 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:27.702678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.3.127.129:39387)
0812 02:41:27.702827 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:27.702831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:27.702846 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:41:27.703009 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:27.703011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:27.703084 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:27.703176 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:41:27.703387 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.703903 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.704838 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:27.705071 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:27.705190 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:27.705552 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:41:27.705999 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:41:27.706099 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:27.706192 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 02:41:27.706248 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 02:41:27.712951 903 raft_consensus.cc:1261] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3 [term 1 FOLLOWER]: Refusing update from remote peer 3cbb0b1c115d45bdbc43f54bfdb680bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:41:27.712951 670 raft_consensus.cc:1261] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762 [term 1 FOLLOWER]: Refusing update from remote peer 3cbb0b1c115d45bdbc43f54bfdb680bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:41:27.713747 1027 consensus_queue.cc:1035] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d16f7d3207b94b6ab4b580c34a9f0762" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 44729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:41:27.713986 1021 consensus_queue.cc:1035] T 4ef6d110bc634a6eb870c8b5a25e4976 P 3cbb0b1c115d45bdbc43f54bfdb680bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1ed7aca49de49478adbba0e83185ff3" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:41:27.723608 1035 mvcc.cc:205] Tried to move back new op lower bound from 7317514189663453184 to 7317514189519081472. Current Snapshot: MvccSnapshot[applied={T|T < 7317514189663453184}]
I20260812 02:41:27.730971 32677 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:41:27.731163 32677 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:41:27.733805 32677 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f98e56c0ffc4fee8bf75f5f8ef1adb3: Sending DeleteTablet for 3 replicas of tablet 4ef6d110bc634a6eb870c8b5a25e4976
I20260812 02:41:27.734802 638 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ef6d110bc634a6eb870c8b5a25e4976 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:41:27 UTC) from {username='slave'} at 127.0.0.1:56320
I20260812 02:41:27.734805 873 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ef6d110bc634a6eb870c8b5a25e4976 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:41:27 UTC) from {username='slave'} at 127.0.0.1:44046
I20260812 02:41:27.735260 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 313
I20260812 02:41:27.736148 1048 tablet_replica.cc:330] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3: stopping tablet replica
I20260812 02:41:27.738888 1048 raft_consensus.cc:2229] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:41:27.739274 1048 raft_consensus.cc:2258] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:41:27.742024 1050 tablet_replica.cc:330] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762: stopping tablet replica
I20260812 02:41:27.742398 1050 raft_consensus.cc:2229] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:41:27.742422 1048 ts_tablet_manager.cc:1905] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:41:27.742770 1050 raft_consensus.cc:2258] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:41:27.744248 1050 ts_tablet_manager.cc:1905] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:41:27.745669 1048 ts_tablet_manager.cc:1918] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:41:27.745795 1048 log.cc:1196] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/4ef6d110bc634a6eb870c8b5a25e4976
I20260812 02:41:27.746134 1048 ts_tablet_manager.cc:1939] T 4ef6d110bc634a6eb870c8b5a25e4976 P a1ed7aca49de49478adbba0e83185ff3: Deleting consensus metadata
I20260812 02:41:27.746769 1050 ts_tablet_manager.cc:1918] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:41:27.746873 1050 log.cc:1196] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/4ef6d110bc634a6eb870c8b5a25e4976
I20260812 02:41:27.747182 1050 ts_tablet_manager.cc:1939] T 4ef6d110bc634a6eb870c8b5a25e4976 P d16f7d3207b94b6ab4b580c34a9f0762: Deleting consensus metadata
I20260812 02:41:27.747733 32661 catalog_manager.cc:4841] TS a1ed7aca49de49478adbba0e83185ff3 (127.3.127.131:45445): tablet 4ef6d110bc634a6eb870c8b5a25e4976 (table test-table [id=70254f8ac4ac49caac26a8ecf85607a9]) successfully deleted
I20260812 02:41:27.748399 32658 catalog_manager.cc:4841] TS d16f7d3207b94b6ab4b580c34a9f0762 (127.3.127.130:44729): tablet 4ef6d110bc634a6eb870c8b5a25e4976 (table test-table [id=70254f8ac4ac49caac26a8ecf85607a9]) successfully deleted
W20260812 02:41:27.750231 32661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.127.129:39387 (error 108)
I20260812 02:41:27.750484 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 534
I20260812 02:41:27.761955 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:27.761369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:39387 (local address 127.0.0.1:43648)
0812 02:41:27.761685 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:27.761812 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.129:39387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208}
W20260812 02:41:27.762441 32661 catalog_manager.cc:4584] TS 3cbb0b1c115d45bdbc43f54bfdb680bd (127.3.127.129:39387): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ef6d110bc634a6eb870c8b5a25e4976: Network error: Client connection negotiation failed: client connection to 127.3.127.129:39387: connect: Connection refused (error 111)
I20260812 02:41:27.773914 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 766
I20260812 02:41:27.793840 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 32633
2026-08-12T02:41:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1444 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:41:27 dist-test-slave-rmhg krb5kdc[1059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:41:27 dist-test-slave-rmhg krb5kdc[1059](info): set up 2 sockets
Aug 12 02:41:27 dist-test-slave-rmhg krb5kdc[1059](info): commencing operation
krb5kdc: starting...
W20260812 02:41:28.973769 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:41:29 dist-test-slave-rmhg krb5kdc[1059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:41:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:41:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:41:29.108134 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:37741
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:37741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:29.228938 1077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:29.229285 1077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:29.229349 1077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:29.233554 1077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:29.233652 1077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:29.233685 1077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:29.233706 1077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:29.233728 1077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:29.239629 1077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:37741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:37741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:29.240882 1077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:29.242031 1077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:29.251215 1092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:29.251289 1086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:29.251379 1077 server_base.cc:1034] running on GCE node
W20260812 02:41:29.251305 1085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:29.252192 1077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:29.253489 1077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:29.254674 1077 hybrid_clock.cc:648] HybridClock initialized: now 1786502489254654 us; error 37 us; skew 500 ppm
Aug 12 02:41:29 dist-test-slave-rmhg krb5kdc[1059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:29.257625 1077 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:41:29.258579 1077 webserver.cc:460] Webserver started at http://127.3.127.190:33775/ using document root <none> and password file <none>
I20260812 02:41:29.259166 1077 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:29.259230 1077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:29.259408 1077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:29.261672 1077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "2a552599654247e0b450cc9922914a85"
format_stamp: "Formatted at 2026-08-12 02:41:29 on dist-test-slave-rmhg"
server_key: "ed21ad53f34644d25c7fccc8849ff6c1"
server_key_iv: "7c2653356b30b78e55a62ce615570519"
server_key_version: "encryptionkey@0"
I20260812 02:41:29.262256 1077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "2a552599654247e0b450cc9922914a85"
format_stamp: "Formatted at 2026-08-12 02:41:29 on dist-test-slave-rmhg"
server_key: "ed21ad53f34644d25c7fccc8849ff6c1"
server_key_iv: "7c2653356b30b78e55a62ce615570519"
server_key_version: "encryptionkey@0"
I20260812 02:41:29.266372 1077 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:41:29.269120 1106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:29.270721 1077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:41:29.270877 1077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "2a552599654247e0b450cc9922914a85"
format_stamp: "Formatted at 2026-08-12 02:41:29 on dist-test-slave-rmhg"
server_key: "ed21ad53f34644d25c7fccc8849ff6c1"
server_key_iv: "7c2653356b30b78e55a62ce615570519"
server_key_version: "encryptionkey@0"
I20260812 02:41:29.270999 1077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:29.294195 1077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:29.297775 1077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:29.297957 1077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:29.305701 1077 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:37741
I20260812 02:41:29.305740 1189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:37741 every 8 connection(s)
I20260812 02:41:29.306838 1077 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:29.310426 1190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:29.316819 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 1077
I20260812 02:41:29.316979 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:41:29.317143 1190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85: Bootstrap starting.
I20260812 02:41:29.317337 3582 external_mini_cluster.cc:1428] Setting key c70b8779d96c6ef87655e6e2aeb5dceb
I20260812 02:41:29.320091 1190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:29.320922 1190 log.cc:824] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:29.323101 1190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85: No bootstrap required, opened a new log
Aug 12 02:41:29 dist-test-slave-rmhg krb5kdc[1059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:41:29.327138 1190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a552599654247e0b450cc9922914a85" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37741 } }
I20260812 02:41:29.327416 1190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:29.327487 1190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a552599654247e0b450cc9922914a85, State: Initialized, Role: FOLLOWER
I20260812 02:41:29.328071 1190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a552599654247e0b450cc9922914a85" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37741 } }
I20260812 02:41:29.328255 1190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:29.328336 1190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:29.328456 1190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:29.329809 1190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a552599654247e0b450cc9922914a85" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37741 } }
I20260812 02:41:29.330197 1190 leader_election.cc:302] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a552599654247e0b450cc9922914a85; no voters:
I20260812 02:41:29.330564 1190 leader_election.cc:288] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:41:29.330708 1198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:29.330960 1198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85 [term 1 LEADER]: Becoming Leader. State: Replica: 2a552599654247e0b450cc9922914a85, State: Running, Role: LEADER
I20260812 02:41:29.332082 1190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:29.332391 1198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a552599654247e0b450cc9922914a85" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37741 } }
I20260812 02:41:29.333938 1195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:29.318901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57386 (local address 127.3.127.190:37741)
0812 02:41:29.319465 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:29.319483 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:29.319539 (+ 56us) server_negotiation.cc:407] Connection header received
0812 02:41:29.320286 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:29.320314 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:29.320756 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:29.321113 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:29.322049 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:29.323109 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:29.324068 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:29.324685 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:29.324851 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:29.327615 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:29.327622 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:29.327640 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:29.327683 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:29.329822 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:29.330424 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:29.330431 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:29.330438 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:29.330608 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:29.331027 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:29.331031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:29.331033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:29.331587 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:29.332167 (+ 580us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:29.332338 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 02:41:29.332679 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":175,"threads_started":1}
I20260812 02:41:29.334301 1203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a552599654247e0b450cc9922914a85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a552599654247e0b450cc9922914a85" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37741 } } }
I20260812 02:41:29.334448 1203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:29.334756 1199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a552599654247e0b450cc9922914a85. Latest consensus state: current_term: 1 leader_uuid: "2a552599654247e0b450cc9922914a85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a552599654247e0b450cc9922914a85" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 37741 } } }
I20260812 02:41:29.334856 1227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:29.336444 1199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:29.339285 1227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:29.345321 1227 catalog_manager.cc:1347] Generated new cluster ID: c80cf400621e40dd85dea53413312fe9
I20260812 02:41:29.345414 1227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:29.357604 1227 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:41:29.358651 1227 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:29.369361 1227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85: Generated new TSK 0
I20260812 02:41:29.370148 1227 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
I20260812 02:41:29.408387 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37741
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:29.530041 1245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:29.530342 1245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:29.530406 1245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:29.534703 1245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:29.534785 1245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:29.534866 1245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:41:29.540663 1245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:29.541903 1245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:29.542997 1245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:29.553495 1256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:29.553529 1259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:29.553495 1257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:29.553555 1245 server_base.cc:1034] running on GCE node
I20260812 02:41:29.554206 1245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:29.554888 1245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:29.556053 1245 hybrid_clock.cc:648] HybridClock initialized: now 1786502489556034 us; error 34 us; skew 500 ppm
Aug 12 02:41:29 dist-test-slave-rmhg krb5kdc[1059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:29.558825 1245 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:41:29.559760 1245 webserver.cc:460] Webserver started at http://127.3.127.129:44409/ using document root <none> and password file <none>
I20260812 02:41:29.560479 1245 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:29.560557 1245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:29.560776 1245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:29.562959 1245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "e700775383884c4880832ebd0dffb9f5"
format_stamp: "Formatted at 2026-08-12 02:41:29 on dist-test-slave-rmhg"
server_key: "3efa09bc0214d55ce4cd4502c288dee9"
server_key_iv: "63a9d80e4862ed52d8fdb35de4e4583e"
server_key_version: "encryptionkey@0"
I20260812 02:41:29.563572 1245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "e700775383884c4880832ebd0dffb9f5"
format_stamp: "Formatted at 2026-08-12 02:41:29 on dist-test-slave-rmhg"
server_key: "3efa09bc0214d55ce4cd4502c288dee9"
server_key_iv: "63a9d80e4862ed52d8fdb35de4e4583e"
server_key_version: "encryptionkey@0"
I20260812 02:41:29.567590 1245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:41:29.570274 1276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:29.571641 1245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:41:29.571787 1245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "e700775383884c4880832ebd0dffb9f5"
format_stamp: "Formatted at 2026-08-12 02:41:29 on dist-test-slave-rmhg"
server_key: "3efa09bc0214d55ce4cd4502c288dee9"
server_key_iv: "63a9d80e4862ed52d8fdb35de4e4583e"
server_key_version: "encryptionkey@0"
I20260812 02:41:29.571906 1245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:29.590302 1245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:29.593839 1245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:29.594044 1245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:29.594754 1245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:41:29.595685 1245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:41:29.595736 1245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:29.595795 1245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:41:29.595822 1245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:29.604977 1245 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:38169
I20260812 02:41:29.605048 1471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:38169 every 8 connection(s)
I20260812 02:41:29.605983 1245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:41:29.606616 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 1245
I20260812 02:41:29.606769 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:41:29.607059 3582 external_mini_cluster.cc:1428] Setting key 14d02396283eff76cee76f28e8a2f4c3
Aug 12 02:41:29 dist-test-slave-rmhg krb5kdc[1059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.127.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.130@KRBTEST.COM" created.
I20260812 02:41:29.620724 1195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:29.607963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:34685 (local address 127.3.127.190:37741)
0812 02:41:29.608099 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:29.608104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:29.609420 (+ 1316us) server_negotiation.cc:407] Connection header received
0812 02:41:29.610879 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:29.610883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:29.610943 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:29.611025 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:29.612307 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:29.612863 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:29.614118 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:29.614362 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:29.614458 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:29.617025 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:29.617029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:29.617031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:29.617058 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:29.618488 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:29.619013 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:29.619016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:29.619018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:29.619099 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:29.619425 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:29.619429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:29.619430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:29.619705 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:29.619799 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:29.620395 (+ 596us) server_negotiation.cc:299] Negotiation successful
0812 02:41:29.620538 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 02:41:29.621654 1476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:29.608536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:37741 (local address 127.3.127.129:34685)
0812 02:41:29.609118 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:29.609307 (+ 189us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:29.610654 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:29.611222 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:29.611230 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:29.611610 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:41:29.612153 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:29.612169 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:29.612999 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:29.613002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:29.613977 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:29.613997 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:29.614518 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:29.614523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:29.614698 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:29.615339 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:41:29.615357 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:41:29.616844 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:41:29.618610 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:29.618617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:29.618636 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:29.618912 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:29.619183 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:29.619186 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:29.619189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:29.619334 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:29.619815 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:41:29.619823 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:41:29.620135 (+ 312us) client_negotiation.cc:769] Sending connection context
0812 02:41:29.620342 (+ 207us) client_negotiation.cc:240] Negotiation successful
0812 02:41:29.620705 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":133,"threads_started":1}
I20260812 02:41:29.622964 1472 heartbeater.cc:347] Connected to a master server at 127.3.127.190:37741
I20260812 02:41:29.623212 1472 heartbeater.cc:464] Registering TS with master...
I20260812 02:41:29.623754 1472 heartbeater.cc:511] Master 127.3.127.190:37741 requested a full tablet report, sending...
I20260812 02:41:29.625746 1134 ts_manager.cc:194] Registered new tserver with Master: e700775383884c4880832ebd0dffb9f5 (127.3.127.129:38169)
I20260812 02:41:29.627405 1134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:34685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
I20260812 02:41:29.637826 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37741
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:29.754581 1480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:29.754864 1480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:29.754907 1480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:29.759052 1480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:29.759117 1480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:29.759176 1480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:41:29.764739 1480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:29.765921 1480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:29.766959 1480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:29.777411 1488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:29.777474 1492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:29.777649 1480 server_base.cc:1034] running on GCE node
W20260812 02:41:29.777448 1489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:29.778214 1480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:29.778889 1480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:29.780097 1480 hybrid_clock.cc:648] HybridClock initialized: now 1786502489780078 us; error 27 us; skew 500 ppm
Aug 12 02:41:29 dist-test-slave-rmhg krb5kdc[1059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:29.783082 1480 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:41:29.784053 1480 webserver.cc:460] Webserver started at http://127.3.127.130:40069/ using document root <none> and password file <none>
I20260812 02:41:29.784673 1480 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:29.784749 1480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:29.784960 1480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:29.787240 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "636d225a5dc9414785ebf687cdb177e2"
format_stamp: "Formatted at 2026-08-12 02:41:29 on dist-test-slave-rmhg"
server_key: "015aabdec99dcab507f6850effdf8ee4"
server_key_iv: "9f9aaa56e0bb88060ddc62d0395696ca"
server_key_version: "encryptionkey@0"
I20260812 02:41:29.787855 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "636d225a5dc9414785ebf687cdb177e2"
format_stamp: "Formatted at 2026-08-12 02:41:29 on dist-test-slave-rmhg"
server_key: "015aabdec99dcab507f6850effdf8ee4"
server_key_iv: "9f9aaa56e0bb88060ddc62d0395696ca"
server_key_version: "encryptionkey@0"
I20260812 02:41:29.791965 1480 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:41:29.795001 1501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:29.796475 1480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:41:29.796620 1480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "636d225a5dc9414785ebf687cdb177e2"
format_stamp: "Formatted at 2026-08-12 02:41:29 on dist-test-slave-rmhg"
server_key: "015aabdec99dcab507f6850effdf8ee4"
server_key_iv: "9f9aaa56e0bb88060ddc62d0395696ca"
server_key_version: "encryptionkey@0"
I20260812 02:41:29.796751 1480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:29.819286 1480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:29.823114 1480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:29.823321 1480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:29.824038 1480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:41:29.825038 1480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:41:29.825091 1480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:29.825133 1480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:41:29.825187 1480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:29.834365 1480 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:33233
I20260812 02:41:29.834412 1700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:33233 every 8 connection(s)
I20260812 02:41:29.835386 1480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:41:29.835717 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 1480
I20260812 02:41:29.835837 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:41:29.836182 3582 external_mini_cluster.cc:1428] Setting key 2b7081f4e3b7e09f2ddcaf24d5f5a4ce
Aug 12 02:41:29 dist-test-slave-rmhg krb5kdc[1059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.127.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.131@KRBTEST.COM" created.
I20260812 02:41:29.851364 1195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:29.837692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:36383 (local address 127.3.127.190:37741)
0812 02:41:29.837812 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:29.837815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:29.838927 (+ 1112us) server_negotiation.cc:407] Connection header received
0812 02:41:29.840605 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:29.840610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:29.840669 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:29.840761 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:29.842051 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:29.842596 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:29.843827 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:29.844051 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:29.844152 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:29.847131 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:29.847135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:29.847137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:29.847178 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:29.848752 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:29.849474 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:29.849479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:29.849481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:29.849596 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:29.849991 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:29.849993 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:29.849994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:29.850229 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:29.850332 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:29.851046 (+ 714us) server_negotiation.cc:299] Negotiation successful
0812 02:41:29.851194 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 02:41:29.852262 1705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:29.838206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:37741 (local address 127.3.127.130:36383)
0812 02:41:29.838741 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:29.838808 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:29.840386 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:29.840919 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:29.840928 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:29.841352 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:41:29.841911 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:29.841927 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:29.842738 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:29.842741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:29.843695 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:29.843707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:29.844208 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:29.844212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:29.844390 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:29.845038 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:41:29.845056 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:41:29.846916 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:41:29.848918 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:29.848931 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:29.848950 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:29.849355 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:29.849700 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:29.849703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:29.849705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:29.849884 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:29.850367 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:41:29.850375 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:41:29.850747 (+ 372us) client_negotiation.cc:769] Sending connection context
0812 02:41:29.850937 (+ 190us) client_negotiation.cc:240] Negotiation successful
0812 02:41:29.851258 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":109,"threads_started":1}
I20260812 02:41:29.853780 1702 heartbeater.cc:347] Connected to a master server at 127.3.127.190:37741
I20260812 02:41:29.854086 1702 heartbeater.cc:464] Registering TS with master...
I20260812 02:41:29.854770 1702 heartbeater.cc:511] Master 127.3.127.190:37741 requested a full tablet report, sending...
I20260812 02:41:29.855991 1134 ts_manager.cc:194] Registered new tserver with Master: 636d225a5dc9414785ebf687cdb177e2 (127.3.127.130:33233)
I20260812 02:41:29.856590 1134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:36383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
I20260812 02:41:29.869450 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37741
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:29.988605 1709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:29.988907 1709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:29.988952 1709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:29.993017 1709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:29.993079 1709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:29.993137 1709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:41:29.998914 1709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:37741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:30.000033 1709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:30.001199 1709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:41:30.011431 1709 server_base.cc:1034] running on GCE node
W20260812 02:41:30.011415 1722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:30.011436 1723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:30.011436 1726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:30.012179 1709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:30.012866 1709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:30.014045 1709 hybrid_clock.cc:648] HybridClock initialized: now 1786502490014006 us; error 53 us; skew 500 ppm
Aug 12 02:41:30 dist-test-slave-rmhg krb5kdc[1059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502490, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:30.016883 1709 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:41:30.017863 1709 webserver.cc:460] Webserver started at http://127.3.127.131:37821/ using document root <none> and password file <none>
I20260812 02:41:30.018453 1709 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:30.018522 1709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:30.018738 1709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:30.020905 1709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "7edb9482cfe5417eb1bc1743da55e810"
format_stamp: "Formatted at 2026-08-12 02:41:30 on dist-test-slave-rmhg"
server_key: "4228d4a95cd4b1e765b590dca30985c1"
server_key_iv: "386eb9f429dbfda5bb21dc4cff9a2239"
server_key_version: "encryptionkey@0"
I20260812 02:41:30.021543 1709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "7edb9482cfe5417eb1bc1743da55e810"
format_stamp: "Formatted at 2026-08-12 02:41:30 on dist-test-slave-rmhg"
server_key: "4228d4a95cd4b1e765b590dca30985c1"
server_key_iv: "386eb9f429dbfda5bb21dc4cff9a2239"
server_key_version: "encryptionkey@0"
I20260812 02:41:30.025748 1709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:41:30.028419 1740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:30.029865 1709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:41:30.029999 1709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "7edb9482cfe5417eb1bc1743da55e810"
format_stamp: "Formatted at 2026-08-12 02:41:30 on dist-test-slave-rmhg"
server_key: "4228d4a95cd4b1e765b590dca30985c1"
server_key_iv: "386eb9f429dbfda5bb21dc4cff9a2239"
server_key_version: "encryptionkey@0"
I20260812 02:41:30.030135 1709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:30.044191 1709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:30.047673 1709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:30.047871 1709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:30.048509 1709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:41:30.049459 1709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:41:30.049512 1709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:30.049571 1709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:41:30.049598 1709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:30.058768 1709 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:39267
I20260812 02:41:30.058805 1929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:39267 every 8 connection(s)
I20260812 02:41:30.059775 1709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:41:30.068079 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 1709
I20260812 02:41:30.068239 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:41:30.068558 3582 external_mini_cluster.cc:1428] Setting key 6802fe8376fe9bcd4f9fbaf68923afeb
Aug 12 02:41:30 dist-test-slave-rmhg krb5kdc[1059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502490, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:41:30.074870 1195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:30.061796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:55285 (local address 127.3.127.190:37741)
0812 02:41:30.061956 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:30.061961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:30.062957 (+ 996us) server_negotiation.cc:407] Connection header received
0812 02:41:30.064335 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:30.064338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:30.064394 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:30.064489 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:30.065903 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.066452 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.067783 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.068126 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.068263 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:30.071387 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:30.071392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:30.071395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:30.071429 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:30.072720 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:30.073334 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:30.073336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:30.073338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:30.073422 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:30.073737 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:30.073738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:30.073740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:30.073929 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:30.074005 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:30.074576 (+ 571us) server_negotiation.cc:299] Negotiation successful
0812 02:41:30.074713 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 02:41:30.076063 1935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:30.062272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:37741 (local address 127.3.127.131:55285)
0812 02:41:30.062783 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:30.062834 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:30.064083 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:30.064677 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:30.064685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:30.065081 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:41:30.065709 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:30.065728 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.066629 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.066632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:30.067590 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:30.067601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.068348 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.068353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:30.068528 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:30.069233 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:41:30.069253 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:41:30.071178 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:41:30.072844 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:30.072850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:30.072867 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:30.073249 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:30.073500 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:30.073503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:30.073504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:30.073662 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:30.074057 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:41:30.074064 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:41:30.074355 (+ 291us) client_negotiation.cc:769] Sending connection context
0812 02:41:30.074615 (+ 260us) client_negotiation.cc:240] Negotiation successful
0812 02:41:30.075018 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":124,"threads_started":1}
I20260812 02:41:30.077414 1931 heartbeater.cc:347] Connected to a master server at 127.3.127.190:37741
I20260812 02:41:30.077697 1931 heartbeater.cc:464] Registering TS with master...
I20260812 02:41:30.078269 1931 heartbeater.cc:511] Master 127.3.127.190:37741 requested a full tablet report, sending...
I20260812 02:41:30.079416 1134 ts_manager.cc:194] Registered new tserver with Master: 7edb9482cfe5417eb1bc1743da55e810 (127.3.127.131:39267)
I20260812 02:41:30.080009 1134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:55285
I20260812 02:41:30.082990 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:41:30.093361 1195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:30.085023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48678 (local address 127.3.127.190:37741)
0812 02:41:30.085220 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:30.085229 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:30.085351 (+ 122us) server_negotiation.cc:407] Connection header received
0812 02:41:30.085479 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:30.085482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:30.085527 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:30.085596 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:30.086436 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.087018 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.087920 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.088155 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.088252 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:30.090098 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:30.090102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:30.090104 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:30.090136 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:30.091355 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:30.091911 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:30.091914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:30.091915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:30.092004 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:30.092339 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:30.092341 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:30.092343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:30.092623 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:30.092772 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:30.093054 (+ 282us) server_negotiation.cc:299] Negotiation successful
0812 02:41:30.093194 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 02:41:30.096918 1134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:41:30.100168 1134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:41:30.115590 1947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:30.109789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44734 (local address 127.3.127.131:39267)
0812 02:41:30.110453 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:30.110469 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:30.110497 (+ 28us) server_negotiation.cc:407] Connection header received
0812 02:41:30.110583 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:30.110588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:30.110812 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:30.110994 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:30.111542 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.112699 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.114314 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.114886 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.114999 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:30.115169 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:30.115333 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 02:41:30.115437 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":106,"threads_started":1}
I20260812 02:41:30.115571 1944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:30.109639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:39267 (local address 127.0.0.1:44734)
0812 02:41:30.110050 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:30.110111 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:30.110251 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:30.111207 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:30.111212 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:30.111238 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:30.111368 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:30.111374 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.112828 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.112831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:30.114197 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:30.114207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.115003 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.115006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:30.115115 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:30.115158 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 02:41:30.115275 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 02:41:30.115407 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":76,"thread_start_us":124,"threads_started":1}
I20260812 02:41:30.115726 1948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:30.109863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:33233 (local address 127.0.0.1:53990)
0812 02:41:30.110748 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:30.110762 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:30.110855 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:30.111600 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:30.111603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:30.111624 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:30.111733 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:30.111741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.112870 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.112873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:30.114230 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:30.114242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.115360 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.115364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:30.115441 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:30.115464 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 02:41:30.115524 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 02:41:30.115582 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":67,"thread_start_us":97,"threads_started":1}
I20260812 02:41:30.116289 1950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:30.110674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.3.127.130:33233)
0812 02:41:30.111085 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:30.111097 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:30.111118 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:41:30.111195 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:30.111201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:30.111408 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:30.111609 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:30.111848 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.112716 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.114374 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.115234 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.115829 (+ 595us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:30.115981 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:30.116036 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 02:41:30.116135 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":119,"threads_started":1}
I20260812 02:41:30.116469 1949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:30.110489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:38169 (local address 127.0.0.1:40568)
0812 02:41:30.111594 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:30.111607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:30.111695 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:30.112206 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:30.112209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:30.112229 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:30.112326 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:30.112333 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.113657 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.113660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:30.114822 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:30.114833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.116049 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.116053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:30.116172 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:30.116197 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 02:41:30.116265 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 02:41:30.116337 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"mutex_wait_us":63,"thread_start_us":97,"threads_started":1}
I20260812 02:41:30.116902 1951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:30.111080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40568 (local address 127.3.127.129:38169)
0812 02:41:30.111482 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:30.111499 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:30.111790 (+ 291us) server_negotiation.cc:407] Connection header received
0812 02:41:30.111871 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:30.111877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:30.112065 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:30.112206 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:30.112439 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.113502 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.114961 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.115763 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.115869 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:30.115990 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:30.116646 (+ 656us) server_negotiation.cc:299] Negotiation successful
0812 02:41:30.116754 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":137,"threads_started":1}
I20260812 02:41:30.119144 1585 tablet_service.cc:1467] Processing CreateTablet for tablet 8c5113817ea34f4494e9dc1bfb514748 (DEFAULT_TABLE table=test-table [id=7bf269ce704f48d794eddf9dcce13d68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:41:30.119275 1818 tablet_service.cc:1467] Processing CreateTablet for tablet 8c5113817ea34f4494e9dc1bfb514748 (DEFAULT_TABLE table=test-table [id=7bf269ce704f48d794eddf9dcce13d68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:41:30.119287 1353 tablet_service.cc:1467] Processing CreateTablet for tablet 8c5113817ea34f4494e9dc1bfb514748 (DEFAULT_TABLE table=test-table [id=7bf269ce704f48d794eddf9dcce13d68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:41:30.120134 1585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c5113817ea34f4494e9dc1bfb514748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:30.120338 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c5113817ea34f4494e9dc1bfb514748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:30.120347 1818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c5113817ea34f4494e9dc1bfb514748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:30.126910 1953 tablet_bootstrap.cc:492] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2: Bootstrap starting.
I20260812 02:41:30.128626 1955 tablet_bootstrap.cc:492] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810: Bootstrap starting.
I20260812 02:41:30.129513 1956 tablet_bootstrap.cc:492] T 8c5113817ea34f4494e9dc1bfb514748 P e700775383884c4880832ebd0dffb9f5: Bootstrap starting.
I20260812 02:41:30.129594 1953 tablet_bootstrap.cc:654] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:30.130472 1953 log.cc:824] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:30.130851 1955 tablet_bootstrap.cc:654] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:30.131695 1956 tablet_bootstrap.cc:654] T 8c5113817ea34f4494e9dc1bfb514748 P e700775383884c4880832ebd0dffb9f5: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:30.132212 1955 log.cc:824] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:30.132499 1956 log.cc:824] T 8c5113817ea34f4494e9dc1bfb514748 P e700775383884c4880832ebd0dffb9f5: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:30.132664 1953 tablet_bootstrap.cc:492] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2: No bootstrap required, opened a new log
I20260812 02:41:30.132889 1953 ts_tablet_manager.cc:1397] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260812 02:41:30.134560 1955 tablet_bootstrap.cc:492] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810: No bootstrap required, opened a new log
I20260812 02:41:30.134560 1956 tablet_bootstrap.cc:492] T 8c5113817ea34f4494e9dc1bfb514748 P e700775383884c4880832ebd0dffb9f5: No bootstrap required, opened a new log
I20260812 02:41:30.134769 1955 ts_tablet_manager.cc:1397] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 02:41:30.134867 1956 ts_tablet_manager.cc:1397] T 8c5113817ea34f4494e9dc1bfb514748 P e700775383884c4880832ebd0dffb9f5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 02:41:30.136133 1953 raft_consensus.cc:348] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7edb9482cfe5417eb1bc1743da55e810" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39267 } } peers { permanent_uuid: "e700775383884c4880832ebd0dffb9f5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 38169 } } peers { permanent_uuid: "636d225a5dc9414785ebf687cdb177e2" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 33233 } }
I20260812 02:41:30.136394 1953 raft_consensus.cc:374] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:30.136451 1953 raft_consensus.cc:729] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 636d225a5dc9414785ebf687cdb177e2, State: Initialized, Role: FOLLOWER
I20260812 02:41:30.136986 1953 consensus_queue.cc:260] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7edb9482cfe5417eb1bc1743da55e810" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39267 } } peers { permanent_uuid: "e700775383884c4880832ebd0dffb9f5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 38169 } } peers { permanent_uuid: "636d225a5dc9414785ebf687cdb177e2" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 33233 } }
I20260812 02:41:30.138064 1953 ts_tablet_manager.cc:1428] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 02:41:30.138442 1702 heartbeater.cc:503] Master 127.3.127.190:37741 was elected leader, sending a full tablet report...
I20260812 02:41:30.138620 1955 raft_consensus.cc:348] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7edb9482cfe5417eb1bc1743da55e810" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39267 } } peers { permanent_uuid: "e700775383884c4880832ebd0dffb9f5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 38169 } } peers { permanent_uuid: "636d225a5dc9414785ebf687cdb177e2" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 33233 } }
I20260812 02:41:30.138921 1955 raft_consensus.cc:374] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:30.139010 1955 raft_consensus.cc:729] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7edb9482cfe5417eb1bc1743da55e810, State: Initialized, Role: FOLLOWER
I20260812 02:41:30.139518 1955 consensus_queue.cc:260] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7edb9482cfe5417eb1bc1743da55e810" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39267 } } peers { permanent_uuid: "e700775383884c4880832ebd0dffb9f5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 38169 } } peers { permanent_uuid: "636d225a5dc9414785ebf687cdb177e2" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 33233 } }
I20260812 02:41:30.140300 1931 heartbeater.cc:503] Master 127.3.127.190:37741 was elected leader, sending a full tablet report...
I20260812 02:41:30.140714 1955 ts_tablet_manager.cc:1428] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 02:41:30.142578 1956 raft_consensus.cc:348] T 8c5113817ea34f4494e9dc1bfb514748 P e700775383884c4880832ebd0dffb9f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7edb9482cfe5417eb1bc1743da55e810" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39267 } } peers { permanent_uuid: "e700775383884c4880832ebd0dffb9f5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 38169 } } peers { permanent_uuid: "636d225a5dc9414785ebf687cdb177e2" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 33233 } }
I20260812 02:41:30.142927 1956 raft_consensus.cc:374] T 8c5113817ea34f4494e9dc1bfb514748 P e700775383884c4880832ebd0dffb9f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:30.143023 1956 raft_consensus.cc:729] T 8c5113817ea34f4494e9dc1bfb514748 P e700775383884c4880832ebd0dffb9f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e700775383884c4880832ebd0dffb9f5, State: Initialized, Role: FOLLOWER
I20260812 02:41:30.143558 1956 consensus_queue.cc:260] T 8c5113817ea34f4494e9dc1bfb514748 P e700775383884c4880832ebd0dffb9f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7edb9482cfe5417eb1bc1743da55e810" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39267 } } peers { permanent_uuid: "e700775383884c4880832ebd0dffb9f5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 38169 } } peers { permanent_uuid: "636d225a5dc9414785ebf687cdb177e2" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 33233 } }
I20260812 02:41:30.144292 1472 heartbeater.cc:503] Master 127.3.127.190:37741 was elected leader, sending a full tablet report...
I20260812 02:41:30.144625 1956 ts_tablet_manager.cc:1428] T 8c5113817ea34f4494e9dc1bfb514748 P e700775383884c4880832ebd0dffb9f5: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
W20260812 02:41:30.310874 1932 tablet.cc:2369] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:41:30.336789 1704 tablet.cc:2369] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:41:30.343523 1965 raft_consensus.cc:482] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:41:30.343816 1965 raft_consensus.cc:504] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7edb9482cfe5417eb1bc1743da55e810" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39267 } } peers { permanent_uuid: "e700775383884c4880832ebd0dffb9f5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 38169 } } peers { permanent_uuid: "636d225a5dc9414785ebf687cdb177e2" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 33233 } }
I20260812 02:41:30.345296 1965 leader_election.cc:288] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e700775383884c4880832ebd0dffb9f5 (127.3.127.129:38169), 636d225a5dc9414785ebf687cdb177e2 (127.3.127.130:33233)
I20260812 02:41:30.349849 1935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:30.345539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:38169 (local address 127.3.127.131:36585)
0812 02:41:30.345702 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:30.345721 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:30.345835 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:30.346117 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:30.346120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:30.346140 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:30.346271 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:30.346279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.347318 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.347321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:30.348457 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:30.348469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.349420 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.349423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:30.349519 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:30.349542 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 02:41:30.349609 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 02:41:30.349687 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 02:41:30.349872 1951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:30.345736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:36585 (local address 127.3.127.129:38169)
0812 02:41:30.345876 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:30.345879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:30.345896 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:41:30.345949 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:30.345951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:30.345994 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:30.346087 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:30.346444 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.347174 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.348663 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.349293 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.349425 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:30.349485 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:30.349655 (+ 170us) server_negotiation.cc:299] Negotiation successful
0812 02:41:30.349739 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 02:41:30.350133 1968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:30.345703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:33233 (local address 127.3.127.131:51403)
0812 02:41:30.346173 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:30.346187 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:30.346269 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:30.346792 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:30.346795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:30.346815 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:30.346939 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:30.346944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.347758 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.347761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:30.348810 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:30.348818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.349740 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.349743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:30.349840 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:30.349865 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 02:41:30.349942 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 02:41:30.350007 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":137,"threads_started":1}
I20260812 02:41:30.350566 1950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:30.346313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:51403 (local address 127.3.127.130:33233)
0812 02:41:30.346475 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:30.346480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:30.346495 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:41:30.346598 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:30.346600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:30.346652 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:30.346735 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:30.347055 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.347616 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.348947 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.349582 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.349702 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:30.349751 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:30.350357 (+ 606us) server_negotiation.cc:299] Negotiation successful
0812 02:41:30.350448 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 02:41:30.350607 1385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5113817ea34f4494e9dc1bfb514748" candidate_uuid: "7edb9482cfe5417eb1bc1743da55e810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e700775383884c4880832ebd0dffb9f5" is_pre_election: true
I20260812 02:41:30.351055 1385 raft_consensus.cc:2454] T 8c5113817ea34f4494e9dc1bfb514748 P e700775383884c4880832ebd0dffb9f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7edb9482cfe5417eb1bc1743da55e810 in term 0.
I20260812 02:41:30.351179 1619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5113817ea34f4494e9dc1bfb514748" candidate_uuid: "7edb9482cfe5417eb1bc1743da55e810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "636d225a5dc9414785ebf687cdb177e2" is_pre_election: true
I20260812 02:41:30.351528 1619 raft_consensus.cc:2454] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7edb9482cfe5417eb1bc1743da55e810 in term 0.
I20260812 02:41:30.351629 1742 leader_election.cc:302] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7edb9482cfe5417eb1bc1743da55e810, e700775383884c4880832ebd0dffb9f5; no voters:
I20260812 02:41:30.351979 1965 raft_consensus.cc:2781] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:41:30.352075 1965 raft_consensus.cc:482] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:41:30.352124 1965 raft_consensus.cc:3037] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:30.353312 1965 raft_consensus.cc:504] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7edb9482cfe5417eb1bc1743da55e810" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39267 } } peers { permanent_uuid: "e700775383884c4880832ebd0dffb9f5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 38169 } } peers { permanent_uuid: "636d225a5dc9414785ebf687cdb177e2" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 33233 } }
I20260812 02:41:30.353665 1965 leader_election.cc:288] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 [CANDIDATE]: Term 1 election: Requested vote from peers e700775383884c4880832ebd0dffb9f5 (127.3.127.129:38169), 636d225a5dc9414785ebf687cdb177e2 (127.3.127.130:33233)
I20260812 02:41:30.354115 1619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5113817ea34f4494e9dc1bfb514748" candidate_uuid: "7edb9482cfe5417eb1bc1743da55e810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "636d225a5dc9414785ebf687cdb177e2"
I20260812 02:41:30.354171 1385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5113817ea34f4494e9dc1bfb514748" candidate_uuid: "7edb9482cfe5417eb1bc1743da55e810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e700775383884c4880832ebd0dffb9f5"
I20260812 02:41:30.354275 1619 raft_consensus.cc:3037] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:30.354321 1385 raft_consensus.cc:3037] T 8c5113817ea34f4494e9dc1bfb514748 P e700775383884c4880832ebd0dffb9f5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:30.355396 1619 raft_consensus.cc:2454] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7edb9482cfe5417eb1bc1743da55e810 in term 1.
I20260812 02:41:30.355443 1385 raft_consensus.cc:2454] T 8c5113817ea34f4494e9dc1bfb514748 P e700775383884c4880832ebd0dffb9f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7edb9482cfe5417eb1bc1743da55e810 in term 1.
I20260812 02:41:30.355821 1744 leader_election.cc:302] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 636d225a5dc9414785ebf687cdb177e2, 7edb9482cfe5417eb1bc1743da55e810, e700775383884c4880832ebd0dffb9f5; no voters:
I20260812 02:41:30.356048 1965 raft_consensus.cc:2781] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:30.356314 1965 raft_consensus.cc:686] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 [term 1 LEADER]: Becoming Leader. State: Replica: 7edb9482cfe5417eb1bc1743da55e810, State: Running, Role: LEADER
I20260812 02:41:30.356719 1965 consensus_queue.cc:237] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7edb9482cfe5417eb1bc1743da55e810" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39267 } } peers { permanent_uuid: "e700775383884c4880832ebd0dffb9f5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 38169 } } peers { permanent_uuid: "636d225a5dc9414785ebf687cdb177e2" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 33233 } }
W20260812 02:41:30.357331 1474 tablet.cc:2369] T 8c5113817ea34f4494e9dc1bfb514748 P e700775383884c4880832ebd0dffb9f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:41:30.360200 1133 catalog_manager.cc:5495] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7edb9482cfe5417eb1bc1743da55e810 (127.3.127.131). New cstate: current_term: 1 leader_uuid: "7edb9482cfe5417eb1bc1743da55e810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7edb9482cfe5417eb1bc1743da55e810" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 39267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e700775383884c4880832ebd0dffb9f5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "636d225a5dc9414785ebf687cdb177e2" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 33233 } health_report { overall_health: UNKNOWN } } }
I20260812 02:41:30.424559 1947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:30.420767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44748 (local address 127.3.127.131:39267)
0812 02:41:30.420941 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:30.420946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:30.420967 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:41:30.421127 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:30.421130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:30.421222 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:30.421321 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:41:30.421594 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.422108 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.422979 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.423194 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.423282 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:30.423622 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:41:30.424086 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:41:30.424227 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:30.424324 (+ 97us) server_negotiation.cc:299] Negotiation successful
0812 02:41:30.424365 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 02:41:30.431715 1385 raft_consensus.cc:1261] T 8c5113817ea34f4494e9dc1bfb514748 P e700775383884c4880832ebd0dffb9f5 [term 1 FOLLOWER]: Refusing update from remote peer 7edb9482cfe5417eb1bc1743da55e810: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:41:30.431774 1619 raft_consensus.cc:1261] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2 [term 1 FOLLOWER]: Refusing update from remote peer 7edb9482cfe5417eb1bc1743da55e810: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:41:30.432605 1969 consensus_queue.cc:1035] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "636d225a5dc9414785ebf687cdb177e2" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 33233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:41:30.432940 1965 consensus_queue.cc:1035] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e700775383884c4880832ebd0dffb9f5" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:41:30.441885 1974 mvcc.cc:205] Tried to move back new op lower bound from 7317514200799162368 to 7317514200504279040. Current Snapshot: MvccSnapshot[applied={T|T < 7317514200799162368}]
I20260812 02:41:30.442130 1976 mvcc.cc:205] Tried to move back new op lower bound from 7317514200799162368 to 7317514200504279040. Current Snapshot: MvccSnapshot[applied={T|T < 7317514200799162368}]
I20260812 02:41:30.453905 1134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:41:30.454138 1134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:41:30.457388 1134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a552599654247e0b450cc9922914a85: Sending DeleteTablet for 3 replicas of tablet 8c5113817ea34f4494e9dc1bfb514748
I20260812 02:41:30.458372 1585 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c5113817ea34f4494e9dc1bfb514748 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:41:30 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:53990
I20260812 02:41:30.458588 1353 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c5113817ea34f4494e9dc1bfb514748 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:41:30 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:40568
I20260812 02:41:30.459079 1987 tablet_replica.cc:330] T 8c5113817ea34f4494e9dc1bfb514748 P e700775383884c4880832ebd0dffb9f5: stopping tablet replica
I20260812 02:41:30.459249 1985 tablet_replica.cc:330] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2: stopping tablet replica
I20260812 02:41:30.459364 1818 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c5113817ea34f4494e9dc1bfb514748 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:41:30 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:44734
I20260812 02:41:30.459511 1987 raft_consensus.cc:2229] T 8c5113817ea34f4494e9dc1bfb514748 P e700775383884c4880832ebd0dffb9f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:41:30.459509 1985 raft_consensus.cc:2229] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:41:30.459903 1985 raft_consensus.cc:2258] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:41:30.459903 1987 raft_consensus.cc:2258] T 8c5113817ea34f4494e9dc1bfb514748 P e700775383884c4880832ebd0dffb9f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:41:30.460597 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 1245
I20260812 02:41:30.476683 1985 ts_tablet_manager.cc:1905] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:41:30.479969 1985 ts_tablet_manager.cc:1918] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:41:30.480113 1985 log.cc:1196] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/8c5113817ea34f4494e9dc1bfb514748
I20260812 02:41:30.480448 1985 ts_tablet_manager.cc:1939] T 8c5113817ea34f4494e9dc1bfb514748 P 636d225a5dc9414785ebf687cdb177e2: Deleting consensus metadata
W20260812 02:41:30.481967 1107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.127.129:38169 (error 108)
I20260812 02:41:30.482033 1109 catalog_manager.cc:4841] TS 636d225a5dc9414785ebf687cdb177e2 (127.3.127.130:33233): tablet 8c5113817ea34f4494e9dc1bfb514748 (table test-table [id=7bf269ce704f48d794eddf9dcce13d68]) successfully deleted
I20260812 02:41:30.482437 1988 tablet_replica.cc:330] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810: stopping tablet replica
I20260812 02:41:30.482843 1988 raft_consensus.cc:2229] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:41:30.483327 1988 raft_consensus.cc:2258] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:41:30.484942 1988 ts_tablet_manager.cc:1905] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:41:30.486785 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 1480
I20260812 02:41:30.488157 1988 ts_tablet_manager.cc:1918] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:41:30.488282 1988 log.cc:1196] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/8c5113817ea34f4494e9dc1bfb514748
I20260812 02:41:30.488641 1988 ts_tablet_manager.cc:1939] T 8c5113817ea34f4494e9dc1bfb514748 P 7edb9482cfe5417eb1bc1743da55e810: Deleting consensus metadata
I20260812 02:41:30.494664 1107 catalog_manager.cc:4841] TS 7edb9482cfe5417eb1bc1743da55e810 (127.3.127.131:39267): tablet 8c5113817ea34f4494e9dc1bfb514748 (table test-table [id=7bf269ce704f48d794eddf9dcce13d68]) successfully deleted
I20260812 02:41:30.495245 1944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:30.494416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:38169 (local address 127.0.0.1:40570)
0812 02:41:30.495019 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:30.495142 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.129:38169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498}
W20260812 02:41:30.495587 1107 catalog_manager.cc:4584] TS e700775383884c4880832ebd0dffb9f5 (127.3.127.129:38169): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8c5113817ea34f4494e9dc1bfb514748: Network error: Client connection negotiation failed: client connection to 127.3.127.129:38169: connect: Connection refused (error 111)
I20260812 02:41:30.502250 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 1709
I20260812 02:41:30.512682 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 1077
2026-08-12T02:41:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2724 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T02:41:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:41:30Z Disabled control of system clock
I20260812 02:41:30.559957 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:41565
--webserver_interface=127.3.127.190
--webserver_port=0
--builtin_ntp_servers=127.3.127.148:46169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:41565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 02:41:30.678925 1994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:30.679243 1994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:30.679299 1994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:30.683528 1994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:30.683612 1994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:30.683635 1994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:30.683657 1994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:30.683691 1994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:30.688768 1994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:46169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:41565
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:41565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:30.690011 1994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:30.691478 1994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:30.700662 2001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:30.700644 2000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:30.700717 2003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:30.701009 1994 server_base.cc:1034] running on GCE node
I20260812 02:41:30.701658 1994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:30.702823 1994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:30.704015 1994 hybrid_clock.cc:648] HybridClock initialized: now 1786502490703983 us; error 46 us; skew 500 ppm
I20260812 02:41:30.706372 1994 webserver.cc:460] Webserver started at http://127.3.127.190:39435/ using document root <none> and password file <none>
I20260812 02:41:30.707055 1994 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:30.707139 1994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:30.707361 1994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:30.709671 1994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "db211b68b67249738cbf959241b929a4"
format_stamp: "Formatted at 2026-08-12 02:41:30 on dist-test-slave-rmhg"
server_key: "0f720d617274e4481ca112c93ce187ab"
server_key_iv: "edcfda85df3b9d58f6f0e1027774a1af"
server_key_version: "encryptionkey@0"
I20260812 02:41:30.710250 1994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "db211b68b67249738cbf959241b929a4"
format_stamp: "Formatted at 2026-08-12 02:41:30 on dist-test-slave-rmhg"
server_key: "0f720d617274e4481ca112c93ce187ab"
server_key_iv: "edcfda85df3b9d58f6f0e1027774a1af"
server_key_version: "encryptionkey@0"
I20260812 02:41:30.714286 1994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:41:30.717233 2011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:30.718649 1994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:41:30.718770 1994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "db211b68b67249738cbf959241b929a4"
format_stamp: "Formatted at 2026-08-12 02:41:30 on dist-test-slave-rmhg"
server_key: "0f720d617274e4481ca112c93ce187ab"
server_key_iv: "edcfda85df3b9d58f6f0e1027774a1af"
server_key_version: "encryptionkey@0"
I20260812 02:41:30.718899 1994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:30.743616 1994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:30.744442 1994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:30.744673 1994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:30.752280 1994 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:41565
I20260812 02:41:30.752303 2104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:41565 every 8 connection(s)
I20260812 02:41:30.753345 1994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:30.756656 2105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:30.757360 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 1994
I20260812 02:41:30.757484 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:41:30.757772 3582 external_mini_cluster.cc:1428] Setting key 2558274b585ece62368b38e316cbad81
I20260812 02:41:30.763757 2105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4: Bootstrap starting.
I20260812 02:41:30.766762 2105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:30.767625 2105 log.cc:824] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:30.768852 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:30.759356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44232 (local address 127.3.127.190:41565)
0812 02:41:30.760048 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:30.760066 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:30.760105 (+ 39us) server_negotiation.cc:407] Connection header received
0812 02:41:30.760798 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:30.760817 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:30.761336 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:30.761875 (+ 539us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:30.762371 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.763715 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.765044 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:30.765621 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:30.765793 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:30.766742 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:30.766748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:30.766763 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:30.767218 (+ 455us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:30.767279 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:30.767301 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:30.767382 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:30.767589 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 02:41:30.767885 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":152,"threads_started":1}
I20260812 02:41:30.769977 2105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4: No bootstrap required, opened a new log
I20260812 02:41:30.773988 2105 raft_consensus.cc:348] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db211b68b67249738cbf959241b929a4" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41565 } }
I20260812 02:41:30.774331 2105 raft_consensus.cc:374] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:30.774377 2105 raft_consensus.cc:729] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db211b68b67249738cbf959241b929a4, State: Initialized, Role: FOLLOWER
I20260812 02:41:30.774811 2105 consensus_queue.cc:260] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db211b68b67249738cbf959241b929a4" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41565 } }
I20260812 02:41:30.774940 2105 raft_consensus.cc:388] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:30.774989 2105 raft_consensus.cc:482] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:30.775081 2105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:30.776079 2105 raft_consensus.cc:504] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db211b68b67249738cbf959241b929a4" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41565 } }
I20260812 02:41:30.776422 2105 leader_election.cc:302] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db211b68b67249738cbf959241b929a4; no voters:
I20260812 02:41:30.776724 2105 leader_election.cc:288] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:41:30.776859 2111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:30.777086 2111 raft_consensus.cc:686] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4 [term 1 LEADER]: Becoming Leader. State: Replica: db211b68b67249738cbf959241b929a4, State: Running, Role: LEADER
I20260812 02:41:30.777472 2111 consensus_queue.cc:237] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db211b68b67249738cbf959241b929a4" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41565 } }
I20260812 02:41:30.778043 2105 sys_catalog.cc:564] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:30.779274 2113 sys_catalog.cc:455] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db211b68b67249738cbf959241b929a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db211b68b67249738cbf959241b929a4" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41565 } } }
I20260812 02:41:30.779325 2116 sys_catalog.cc:455] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db211b68b67249738cbf959241b929a4. Latest consensus state: current_term: 1 leader_uuid: "db211b68b67249738cbf959241b929a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db211b68b67249738cbf959241b929a4" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41565 } } }
I20260812 02:41:30.779410 2113 sys_catalog.cc:458] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:30.779425 2116 sys_catalog.cc:458] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4 [sys.catalog]: This master's current role is: LEADER
W20260812 02:41:30.783003 2132 catalog_manager.cc:1558] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:41:30.783087 2132 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:41:30.783170 2124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:30.783892 2124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:30.790127 2124 catalog_manager.cc:1347] Generated new cluster ID: 49ea806cc08744f98a245dd54c9f1446
I20260812 02:41:30.790225 2124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:30.800736 2124 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:41:30.802007 2124 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:30.820418 2124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4: Generated new TSK 0
I20260812 02:41:30.821456 2124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:41:30.836418 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:41565
--builtin_ntp_servers=127.3.127.148:46169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 02:41:30.961740 2136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:30.962081 2136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:30.962123 2136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:30.966617 2136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:30.966701 2136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:30.966760 2136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:41:30.971949 2136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:46169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:41565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:30.973285 2136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:30.974833 2136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:41:30.985507 2136 server_base.cc:1034] running on GCE node
W20260812 02:41:30.985402 2142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:30.985402 2145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:30.985419 2143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:30.986250 2136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:30.986963 2136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:30.988183 2136 hybrid_clock.cc:648] HybridClock initialized: now 1786502490988162 us; error 36 us; skew 500 ppm
I20260812 02:41:30.990399 2136 webserver.cc:460] Webserver started at http://127.3.127.129:34581/ using document root <none> and password file <none>
I20260812 02:41:30.991039 2136 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:30.991119 2136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:30.991340 2136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:30.993579 2136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "050d9b44f01e4abc93dfb133af08e3dd"
format_stamp: "Formatted at 2026-08-12 02:41:30 on dist-test-slave-rmhg"
server_key: "a44f47defb116eb1762f66baba2ccb35"
server_key_iv: "a4d99f4592b77c7c03de63543f34480e"
server_key_version: "encryptionkey@0"
I20260812 02:41:30.994153 2136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "050d9b44f01e4abc93dfb133af08e3dd"
format_stamp: "Formatted at 2026-08-12 02:41:30 on dist-test-slave-rmhg"
server_key: "a44f47defb116eb1762f66baba2ccb35"
server_key_iv: "a4d99f4592b77c7c03de63543f34480e"
server_key_version: "encryptionkey@0"
I20260812 02:41:30.998052 2136 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:41:31.000600 2158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:31.001911 2136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 02:41:31.002036 2136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "050d9b44f01e4abc93dfb133af08e3dd"
format_stamp: "Formatted at 2026-08-12 02:41:30 on dist-test-slave-rmhg"
server_key: "a44f47defb116eb1762f66baba2ccb35"
server_key_iv: "a4d99f4592b77c7c03de63543f34480e"
server_key_version: "encryptionkey@0"
I20260812 02:41:31.002156 2136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:31.017773 2136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:31.018560 2136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:31.018751 2136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:31.019398 2136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:41:31.020433 2136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:41:31.020483 2136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:31.020539 2136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:41:31.020568 2136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:31.030054 2136 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:43637
I20260812 02:41:31.030122 2364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:43637 every 8 connection(s)
I20260812 02:41:31.031046 2136 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:41:31.035602 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 2136
I20260812 02:41:31.035722 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:41:31.035991 3582 external_mini_cluster.cc:1428] Setting key 8e656df4d13b449b5c054c909006e11f
I20260812 02:41:31.038255 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:41565
--builtin_ntp_servers=127.3.127.148:46169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 02:41:31.041857 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:31.033092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:40143 (local address 127.3.127.190:41565)
0812 02:41:31.033277 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:31.033281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:31.034464 (+ 1183us) server_negotiation.cc:407] Connection header received
0812 02:41:31.035965 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:31.035969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:31.036039 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:31.036116 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:31.037322 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.037966 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.039671 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.039911 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.040048 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:31.041297 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:31.041300 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:31.041302 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:31.041317 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:31.041337 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:31.041346 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:31.041409 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:31.041578 (+ 169us) server_negotiation.cc:299] Negotiation successful
0812 02:41:31.041696 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 02:41:31.042780 2368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:31.033573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41565 (local address 127.3.127.129:40143)
0812 02:41:31.034299 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:31.034344 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:31.035731 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:31.036365 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:31.036373 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:31.036479 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:41:31.037134 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:31.037181 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.038154 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.038158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:31.039526 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:31.039540 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.040104 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.040108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:31.040302 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:31.040955 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:41:31.040957 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:41:31.041005 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:41:31.041083 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:41:31.041419 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:41:31.041432 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:41:31.041438 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 02:41:31.041556 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 02:41:31.041897 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":137,"threads_started":1}
I20260812 02:41:31.044096 2365 heartbeater.cc:347] Connected to a master server at 127.3.127.190:41565
I20260812 02:41:31.044349 2365 heartbeater.cc:464] Registering TS with master...
I20260812 02:41:31.044934 2365 heartbeater.cc:511] Master 127.3.127.190:41565 requested a full tablet report, sending...
I20260812 02:41:31.046787 2040 ts_manager.cc:194] Registered new tserver with Master: 050d9b44f01e4abc93dfb133af08e3dd (127.3.127.129:43637)
I20260812 02:41:31.048487 2040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.127.129:40143
W20260812 02:41:31.164790 2370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:31.165105 2370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:31.165175 2370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:31.169826 2370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:31.169909 2370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:31.169970 2370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:41:31.176146 2370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:46169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:41565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:31.177461 2370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:31.179054 2370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:41:31.189760 2370 server_base.cc:1034] running on GCE node
W20260812 02:41:31.189692 2376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:31.189669 2375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:31.189669 2379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:31.190420 2370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:31.191150 2370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:31.192325 2370 hybrid_clock.cc:648] HybridClock initialized: now 1786502491192283 us; error 55 us; skew 500 ppm
I20260812 02:41:31.194404 2370 webserver.cc:460] Webserver started at http://127.3.127.130:45021/ using document root <none> and password file <none>
I20260812 02:41:31.195013 2370 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:31.195084 2370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:31.195309 2370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:31.197548 2370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "5b6c74b73eeb412887a9e933f11212c9"
format_stamp: "Formatted at 2026-08-12 02:41:31 on dist-test-slave-rmhg"
server_key: "cc6b7561716ecfe9e20bab58a556952a"
server_key_iv: "714caaaf219d0c98b9acd4a0371bc6fc"
server_key_version: "encryptionkey@0"
I20260812 02:41:31.198136 2370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "5b6c74b73eeb412887a9e933f11212c9"
format_stamp: "Formatted at 2026-08-12 02:41:31 on dist-test-slave-rmhg"
server_key: "cc6b7561716ecfe9e20bab58a556952a"
server_key_iv: "714caaaf219d0c98b9acd4a0371bc6fc"
server_key_version: "encryptionkey@0"
I20260812 02:41:31.202119 2370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:41:31.204777 2387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:31.206280 2370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:41:31.206419 2370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "5b6c74b73eeb412887a9e933f11212c9"
format_stamp: "Formatted at 2026-08-12 02:41:31 on dist-test-slave-rmhg"
server_key: "cc6b7561716ecfe9e20bab58a556952a"
server_key_iv: "714caaaf219d0c98b9acd4a0371bc6fc"
server_key_version: "encryptionkey@0"
I20260812 02:41:31.206575 2370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:31.226614 2370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:31.227443 2370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:31.227648 2370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:31.228318 2370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:41:31.229449 2370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:41:31.229503 2370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:31.229565 2370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:41:31.229591 2370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:31.238926 2370 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:38173
I20260812 02:41:31.238999 2579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:38173 every 8 connection(s)
I20260812 02:41:31.239982 2370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:41:31.247285 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 2370
I20260812 02:41:31.247432 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:41:31.247715 3582 external_mini_cluster.cc:1428] Setting key e6415f4b5b44e5c3c82181728f7cbf00
I20260812 02:41:31.250026 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:41565
--builtin_ntp_servers=127.3.127.148:46169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 02:41:31.250969 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:31.242147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:51313 (local address 127.3.127.190:41565)
0812 02:41:31.242327 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:31.242331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:31.243568 (+ 1237us) server_negotiation.cc:407] Connection header received
0812 02:41:31.245120 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:31.245124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:31.245218 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:31.245300 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:31.246384 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.246933 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.248556 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.248872 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.249007 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:31.250368 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:31.250373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:31.250375 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:31.250394 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:31.250420 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:31.250431 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:31.250519 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:31.250729 (+ 210us) server_negotiation.cc:299] Negotiation successful
0812 02:41:31.250811 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 02:41:31.251978 2582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:31.242608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41565 (local address 127.3.127.130:51313)
0812 02:41:31.243385 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:31.243434 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:31.244860 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:31.245523 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:31.245531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:31.245631 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:41:31.246214 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:31.246230 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.247119 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.247124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:31.248240 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:31.248254 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.249085 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.249090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:31.249334 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:31.249988 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:41:31.249989 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:41:31.250044 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:41:31.250130 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:41:31.250520 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:41:31.250529 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:41:31.250534 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 02:41:31.250678 (+ 144us) client_negotiation.cc:240] Negotiation successful
0812 02:41:31.250979 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":143,"threads_started":1}
I20260812 02:41:31.253383 2580 heartbeater.cc:347] Connected to a master server at 127.3.127.190:41565
I20260812 02:41:31.253706 2580 heartbeater.cc:464] Registering TS with master...
I20260812 02:41:31.254344 2580 heartbeater.cc:511] Master 127.3.127.190:41565 requested a full tablet report, sending...
I20260812 02:41:31.255578 2040 ts_manager.cc:194] Registered new tserver with Master: 5b6c74b73eeb412887a9e933f11212c9 (127.3.127.130:38173)
I20260812 02:41:31.256217 2040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.127.130:51313
W20260812 02:41:31.372936 2583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:31.373281 2583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:31.373345 2583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:31.377769 2583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:31.377839 2583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:31.377920 2583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:41:31.383219 2583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:46169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:41565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:31.384477 2583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:31.386037 2583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:41:31.396682 2583 server_base.cc:1034] running on GCE node
W20260812 02:41:31.396620 2592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:31.396617 2599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:31.396644 2591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:31.397459 2583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:31.398231 2583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:31.399427 2583 hybrid_clock.cc:648] HybridClock initialized: now 1786502491399404 us; error 37 us; skew 500 ppm
I20260812 02:41:31.401558 2583 webserver.cc:460] Webserver started at http://127.3.127.131:44025/ using document root <none> and password file <none>
I20260812 02:41:31.402186 2583 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:31.402262 2583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:31.402484 2583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:31.404726 2583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "b85b0ea54a6040a6939bb54cf072234a"
format_stamp: "Formatted at 2026-08-12 02:41:31 on dist-test-slave-rmhg"
server_key: "e78475b5269e0e72c54d865dfe9ea444"
server_key_iv: "ac2155f36277184bb3fbd73737e2a8d3"
server_key_version: "encryptionkey@0"
I20260812 02:41:31.405349 2583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "b85b0ea54a6040a6939bb54cf072234a"
format_stamp: "Formatted at 2026-08-12 02:41:31 on dist-test-slave-rmhg"
server_key: "e78475b5269e0e72c54d865dfe9ea444"
server_key_iv: "ac2155f36277184bb3fbd73737e2a8d3"
server_key_version: "encryptionkey@0"
I20260812 02:41:31.409469 2583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:41:31.412276 2608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:31.413686 2583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:41:31.413816 2583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "b85b0ea54a6040a6939bb54cf072234a"
format_stamp: "Formatted at 2026-08-12 02:41:31 on dist-test-slave-rmhg"
server_key: "e78475b5269e0e72c54d865dfe9ea444"
server_key_iv: "ac2155f36277184bb3fbd73737e2a8d3"
server_key_version: "encryptionkey@0"
I20260812 02:41:31.413933 2583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:31.428558 2583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:31.429386 2583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:31.429589 2583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:31.430233 2583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:41:31.431262 2583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:41:31.431314 2583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:31.431370 2583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:41:31.431398 2583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:31.439998 2583 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:42505
I20260812 02:41:31.440047 2809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:42505 every 8 connection(s)
I20260812 02:41:31.440944 2583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:41:31.448844 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 2583
I20260812 02:41:31.449007 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:41:31.449354 3582 external_mini_cluster.cc:1428] Setting key cdae5f9f0cb42458ef67ac77d4b48e6e
I20260812 02:41:31.451231 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:31.442952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:53419 (local address 127.3.127.190:41565)
0812 02:41:31.443088 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:31.443092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:31.444253 (+ 1161us) server_negotiation.cc:407] Connection header received
0812 02:41:31.445862 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:31.445866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:31.445936 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:31.446023 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:31.447057 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.447606 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.449033 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.449350 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.449472 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:31.450733 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:31.450737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:31.450740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:31.450754 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:31.450779 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:31.450791 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:31.450860 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:31.451039 (+ 179us) server_negotiation.cc:299] Negotiation successful
0812 02:41:31.451108 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 02:41:31.452317 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:31.443373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41565 (local address 127.3.127.131:53419)
0812 02:41:31.444082 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:31.444127 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:31.445609 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:31.446247 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:31.446255 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:31.446357 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:41:31.446899 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:31.446913 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.447757 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.447760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:31.448846 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:31.448860 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.449556 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.449561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:31.449776 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:31.450426 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:41:31.450428 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:41:31.450493 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:41:31.450594 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:41:31.450887 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:41:31.450897 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:41:31.450904 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 02:41:31.451044 (+ 140us) client_negotiation.cc:240] Negotiation successful
0812 02:41:31.451402 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":139,"threads_started":1}
I20260812 02:41:31.453533 2810 heartbeater.cc:347] Connected to a master server at 127.3.127.190:41565
I20260812 02:41:31.453791 2810 heartbeater.cc:464] Registering TS with master...
I20260812 02:41:31.454396 2810 heartbeater.cc:511] Master 127.3.127.190:41565 requested a full tablet report, sending...
I20260812 02:41:31.455595 2040 ts_manager.cc:194] Registered new tserver with Master: b85b0ea54a6040a6939bb54cf072234a (127.3.127.131:42505)
I20260812 02:41:31.456225 2040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.127.131:53419
I20260812 02:41:31.464094 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:41:31.470398 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:31.466348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44244 (local address 127.3.127.190:41565)
0812 02:41:31.466535 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:31.466539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:31.466659 (+ 120us) server_negotiation.cc:407] Connection header received
0812 02:41:31.466777 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:31.466780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:31.466825 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:31.466895 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:31.467197 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.467767 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.468705 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.468940 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.469021 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:31.469909 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:31.469912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:31.469915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:31.469929 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:31.469949 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:31.469957 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:31.470070 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:31.470195 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 02:41:31.470236 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 02:41:31.473747 2040 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:41:31.476903 2040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:41:31.493289 2832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:31.487251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47964 (local address 127.3.127.130:38173)
0812 02:41:31.488323 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:31.488344 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:31.488371 (+ 27us) server_negotiation.cc:407] Connection header received
0812 02:41:31.488453 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:31.488458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:31.488718 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:31.489011 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:31.489423 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.490529 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.491978 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.492554 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.492658 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:31.492881 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:31.492982 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 02:41:31.493096 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":955,"thread_start_us":131,"threads_started":1}
I20260812 02:41:31.493283 2826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:31.486439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:38173 (local address 127.0.0.1:47964)
0812 02:41:31.487496 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:31.487509 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:31.487601 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:31.489009 (+ 1408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:31.489016 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:31.489048 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:31.489271 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:31.489279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.490633 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.490637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:31.491859 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:31.491869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.492676 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.492679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:31.492789 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:31.492828 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 02:41:31.492950 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 02:41:31.493081 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"thread_start_us":146,"threads_started":1}
I20260812 02:41:31.494474 2827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:31.486361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:43637 (local address 127.0.0.1:60284)
0812 02:41:31.487103 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:31.487343 (+ 240us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:31.487511 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:31.489442 (+ 1931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:31.489446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:31.489480 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:31.489659 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:31.489696 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.491622 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.491626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:31.492950 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:31.492969 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.494044 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.494048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:31.494139 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:31.494166 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 02:41:31.494233 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 02:41:31.494332 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":27,"thread_start_us":105,"threads_started":1}
I20260812 02:41:31.494565 2834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:31.487960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60284 (local address 127.3.127.129:43637)
0812 02:41:31.488654 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:31.488668 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:31.488725 (+ 57us) server_negotiation.cc:407] Connection header received
0812 02:41:31.488843 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:31.488873 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:31.489191 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:31.489915 (+ 724us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:31.490163 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.491490 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.493076 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.493902 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.494017 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:31.494211 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:31.494265 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 02:41:31.494367 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":174,"threads_started":1}
I20260812 02:41:31.495949 2830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:31.487186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:42505 (local address 127.0.0.1:51224)
0812 02:41:31.489303 (+ 2117us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:31.489316 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:31.489444 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:31.490125 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:31.490128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:31.490146 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:31.490235 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:31.490241 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.491622 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.491626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:31.493108 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:31.493115 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.494835 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.494839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:31.495620 (+ 781us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:31.495648 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 02:41:31.495727 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 02:41:31.495805 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2053,"mutex_wait_us":135,"thread_start_us":99,"threads_started":1}
I20260812 02:41:31.496403 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:31.487695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51224 (local address 127.3.127.131:42505)
0812 02:41:31.487995 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:31.488007 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:31.489557 (+ 1550us) server_negotiation.cc:407] Connection header received
0812 02:41:31.489639 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:31.489644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:31.489817 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:31.489967 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:31.490370 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.491490 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.493276 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.494383 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.494488 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:31.494996 (+ 508us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:31.496139 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0812 02:41:31.496244 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":103,"threads_started":1}
I20260812 02:41:31.496682 2470 tablet_service.cc:1467] Processing CreateTablet for tablet 561f70a5ae04495c989740f6167ec8e4 (DEFAULT_TABLE table=test-table [id=88663976e0494795af91c1654af04f03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:41:31.497864 2470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561f70a5ae04495c989740f6167ec8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:31.497635 2246 tablet_service.cc:1467] Processing CreateTablet for tablet 561f70a5ae04495c989740f6167ec8e4 (DEFAULT_TABLE table=test-table [id=88663976e0494795af91c1654af04f03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:41:31.498567 2246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561f70a5ae04495c989740f6167ec8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:31.499233 2696 tablet_service.cc:1467] Processing CreateTablet for tablet 561f70a5ae04495c989740f6167ec8e4 (DEFAULT_TABLE table=test-table [id=88663976e0494795af91c1654af04f03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:41:31.500212 2696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561f70a5ae04495c989740f6167ec8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:31.504755 2835 tablet_bootstrap.cc:492] T 561f70a5ae04495c989740f6167ec8e4 P 050d9b44f01e4abc93dfb133af08e3dd: Bootstrap starting.
I20260812 02:41:31.506425 2836 tablet_bootstrap.cc:492] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9: Bootstrap starting.
I20260812 02:41:31.508163 2835 tablet_bootstrap.cc:654] T 561f70a5ae04495c989740f6167ec8e4 P 050d9b44f01e4abc93dfb133af08e3dd: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:31.508195 2837 tablet_bootstrap.cc:492] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a: Bootstrap starting.
I20260812 02:41:31.508756 2836 tablet_bootstrap.cc:654] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:31.509208 2835 log.cc:824] T 561f70a5ae04495c989740f6167ec8e4 P 050d9b44f01e4abc93dfb133af08e3dd: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:31.509647 2836 log.cc:824] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:31.510345 2837 tablet_bootstrap.cc:654] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:31.511181 2837 log.cc:824] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:31.511411 2835 tablet_bootstrap.cc:492] T 561f70a5ae04495c989740f6167ec8e4 P 050d9b44f01e4abc93dfb133af08e3dd: No bootstrap required, opened a new log
I20260812 02:41:31.511627 2835 ts_tablet_manager.cc:1397] T 561f70a5ae04495c989740f6167ec8e4 P 050d9b44f01e4abc93dfb133af08e3dd: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 02:41:31.512032 2836 tablet_bootstrap.cc:492] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9: No bootstrap required, opened a new log
I20260812 02:41:31.512234 2836 ts_tablet_manager.cc:1397] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 02:41:31.513345 2837 tablet_bootstrap.cc:492] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a: No bootstrap required, opened a new log
I20260812 02:41:31.513559 2837 ts_tablet_manager.cc:1397] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 02:41:31.515507 2835 raft_consensus.cc:348] T 561f70a5ae04495c989740f6167ec8e4 P 050d9b44f01e4abc93dfb133af08e3dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "050d9b44f01e4abc93dfb133af08e3dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43637 } } peers { permanent_uuid: "5b6c74b73eeb412887a9e933f11212c9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 38173 } } peers { permanent_uuid: "b85b0ea54a6040a6939bb54cf072234a" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42505 } }
I20260812 02:41:31.515820 2835 raft_consensus.cc:374] T 561f70a5ae04495c989740f6167ec8e4 P 050d9b44f01e4abc93dfb133af08e3dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:31.515882 2835 raft_consensus.cc:729] T 561f70a5ae04495c989740f6167ec8e4 P 050d9b44f01e4abc93dfb133af08e3dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 050d9b44f01e4abc93dfb133af08e3dd, State: Initialized, Role: FOLLOWER
I20260812 02:41:31.516386 2835 consensus_queue.cc:260] T 561f70a5ae04495c989740f6167ec8e4 P 050d9b44f01e4abc93dfb133af08e3dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "050d9b44f01e4abc93dfb133af08e3dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43637 } } peers { permanent_uuid: "5b6c74b73eeb412887a9e933f11212c9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 38173 } } peers { permanent_uuid: "b85b0ea54a6040a6939bb54cf072234a" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42505 } }
I20260812 02:41:31.516752 2836 raft_consensus.cc:348] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "050d9b44f01e4abc93dfb133af08e3dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43637 } } peers { permanent_uuid: "5b6c74b73eeb412887a9e933f11212c9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 38173 } } peers { permanent_uuid: "b85b0ea54a6040a6939bb54cf072234a" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42505 } }
I20260812 02:41:31.517071 2836 raft_consensus.cc:374] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:31.517136 2836 raft_consensus.cc:729] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b6c74b73eeb412887a9e933f11212c9, State: Initialized, Role: FOLLOWER
I20260812 02:41:31.517199 2365 heartbeater.cc:503] Master 127.3.127.190:41565 was elected leader, sending a full tablet report...
I20260812 02:41:31.517665 2836 consensus_queue.cc:260] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "050d9b44f01e4abc93dfb133af08e3dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43637 } } peers { permanent_uuid: "5b6c74b73eeb412887a9e933f11212c9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 38173 } } peers { permanent_uuid: "b85b0ea54a6040a6939bb54cf072234a" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42505 } }
I20260812 02:41:31.518064 2837 raft_consensus.cc:348] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "050d9b44f01e4abc93dfb133af08e3dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43637 } } peers { permanent_uuid: "5b6c74b73eeb412887a9e933f11212c9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 38173 } } peers { permanent_uuid: "b85b0ea54a6040a6939bb54cf072234a" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42505 } }
I20260812 02:41:31.518338 2837 raft_consensus.cc:374] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:31.518388 2837 raft_consensus.cc:729] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b85b0ea54a6040a6939bb54cf072234a, State: Initialized, Role: FOLLOWER
I20260812 02:41:31.518649 2580 heartbeater.cc:503] Master 127.3.127.190:41565 was elected leader, sending a full tablet report...
I20260812 02:41:31.518854 2836 ts_tablet_manager.cc:1428] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 02:41:31.519281 2835 ts_tablet_manager.cc:1428] T 561f70a5ae04495c989740f6167ec8e4 P 050d9b44f01e4abc93dfb133af08e3dd: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20260812 02:41:31.519378 2837 consensus_queue.cc:260] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "050d9b44f01e4abc93dfb133af08e3dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43637 } } peers { permanent_uuid: "5b6c74b73eeb412887a9e933f11212c9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 38173 } } peers { permanent_uuid: "b85b0ea54a6040a6939bb54cf072234a" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42505 } }
I20260812 02:41:31.520175 2810 heartbeater.cc:503] Master 127.3.127.190:41565 was elected leader, sending a full tablet report...
I20260812 02:41:31.520376 2837 ts_tablet_manager.cc:1428] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
W20260812 02:41:31.532330 2367 tablet.cc:2369] T 561f70a5ae04495c989740f6167ec8e4 P 050d9b44f01e4abc93dfb133af08e3dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:41:31.673878 2853 raft_consensus.cc:482] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:41:31.674103 2853 raft_consensus.cc:504] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "050d9b44f01e4abc93dfb133af08e3dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43637 } } peers { permanent_uuid: "5b6c74b73eeb412887a9e933f11212c9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 38173 } } peers { permanent_uuid: "b85b0ea54a6040a6939bb54cf072234a" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42505 } }
I20260812 02:41:31.675275 2853 leader_election.cc:288] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 050d9b44f01e4abc93dfb133af08e3dd (127.3.127.129:43637), 5b6c74b73eeb412887a9e933f11212c9 (127.3.127.130:38173)
I20260812 02:41:31.679934 2834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:31.675674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:34491 (local address 127.3.127.129:43637)
0812 02:41:31.675824 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:31.675829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:31.675846 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:41:31.675902 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:31.675905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:31.675957 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:31.676044 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:31.676408 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.677129 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.678689 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.679331 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.679461 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:31.679516 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:31.679738 (+ 222us) server_negotiation.cc:299] Negotiation successful
0812 02:41:31.679810 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 02:41:31.679967 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:31.675566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:43637 (local address 127.3.127.131:34491)
0812 02:41:31.675700 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:31.675716 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:31.675823 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:31.676102 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:31.676105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:31.676134 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:31.676265 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:31.676274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.677290 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.677293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:31.678542 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:31.678555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.679476 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.679479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:31.679570 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:31.679595 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 02:41:31.679712 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 02:41:31.679798 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 02:41:31.680172 2832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:31.675701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:54349 (local address 127.3.127.130:38173)
0812 02:41:31.675825 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:31.675829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:31.676431 (+ 602us) server_negotiation.cc:407] Connection header received
0812 02:41:31.676495 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:31.676498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:31.676593 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:31.676676 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:31.676894 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.677514 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.678948 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.679582 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.679686 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:31.679734 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:31.679967 (+ 233us) server_negotiation.cc:299] Negotiation successful
0812 02:41:31.680052 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 02:41:31.680356 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:31.675589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:38173 (local address 127.3.127.131:54349)
0812 02:41:31.676324 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:31.676337 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:31.676429 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:31.676684 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:31.676686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:31.676697 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:31.676804 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:31.676808 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.677662 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.677669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:31.678793 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:31.678801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.679720 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.679723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:31.679818 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:31.679838 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 02:41:31.679909 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 02:41:31.680171 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":301,"thread_start_us":152,"threads_started":1}
I20260812 02:41:31.680778 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561f70a5ae04495c989740f6167ec8e4" candidate_uuid: "b85b0ea54a6040a6939bb54cf072234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "050d9b44f01e4abc93dfb133af08e3dd" is_pre_election: true
I20260812 02:41:31.681017 2506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561f70a5ae04495c989740f6167ec8e4" candidate_uuid: "b85b0ea54a6040a6939bb54cf072234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b6c74b73eeb412887a9e933f11212c9" is_pre_election: true
I20260812 02:41:31.681126 2279 raft_consensus.cc:2454] T 561f70a5ae04495c989740f6167ec8e4 P 050d9b44f01e4abc93dfb133af08e3dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b85b0ea54a6040a6939bb54cf072234a in term 0.
I20260812 02:41:31.681427 2506 raft_consensus.cc:2454] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b85b0ea54a6040a6939bb54cf072234a in term 0.
I20260812 02:41:31.681743 2617 leader_election.cc:302] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 050d9b44f01e4abc93dfb133af08e3dd, b85b0ea54a6040a6939bb54cf072234a; no voters:
I20260812 02:41:31.682060 2853 raft_consensus.cc:2781] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:41:31.682405 2853 raft_consensus.cc:482] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:41:31.682471 2853 raft_consensus.cc:3037] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:31.683519 2853 raft_consensus.cc:504] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "050d9b44f01e4abc93dfb133af08e3dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43637 } } peers { permanent_uuid: "5b6c74b73eeb412887a9e933f11212c9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 38173 } } peers { permanent_uuid: "b85b0ea54a6040a6939bb54cf072234a" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42505 } }
I20260812 02:41:31.683904 2853 leader_election.cc:288] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a [CANDIDATE]: Term 1 election: Requested vote from peers 050d9b44f01e4abc93dfb133af08e3dd (127.3.127.129:43637), 5b6c74b73eeb412887a9e933f11212c9 (127.3.127.130:38173)
I20260812 02:41:31.684319 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561f70a5ae04495c989740f6167ec8e4" candidate_uuid: "b85b0ea54a6040a6939bb54cf072234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "050d9b44f01e4abc93dfb133af08e3dd"
I20260812 02:41:31.684319 2506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561f70a5ae04495c989740f6167ec8e4" candidate_uuid: "b85b0ea54a6040a6939bb54cf072234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b6c74b73eeb412887a9e933f11212c9"
I20260812 02:41:31.684474 2506 raft_consensus.cc:3037] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:31.684474 2279 raft_consensus.cc:3037] T 561f70a5ae04495c989740f6167ec8e4 P 050d9b44f01e4abc93dfb133af08e3dd [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:31.685907 2506 raft_consensus.cc:2454] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b85b0ea54a6040a6939bb54cf072234a in term 1.
I20260812 02:41:31.685907 2279 raft_consensus.cc:2454] T 561f70a5ae04495c989740f6167ec8e4 P 050d9b44f01e4abc93dfb133af08e3dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b85b0ea54a6040a6939bb54cf072234a in term 1.
I20260812 02:41:31.686259 2617 leader_election.cc:302] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 050d9b44f01e4abc93dfb133af08e3dd, b85b0ea54a6040a6939bb54cf072234a; no voters:
I20260812 02:41:31.686470 2853 raft_consensus.cc:2781] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:31.686743 2853 raft_consensus.cc:686] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a [term 1 LEADER]: Becoming Leader. State: Replica: b85b0ea54a6040a6939bb54cf072234a, State: Running, Role: LEADER
I20260812 02:41:31.687103 2853 consensus_queue.cc:237] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "050d9b44f01e4abc93dfb133af08e3dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43637 } } peers { permanent_uuid: "5b6c74b73eeb412887a9e933f11212c9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 38173 } } peers { permanent_uuid: "b85b0ea54a6040a6939bb54cf072234a" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42505 } }
I20260812 02:41:31.690472 2032 catalog_manager.cc:5495] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a reported cstate change: term changed from 0 to 1, leader changed from <none> to b85b0ea54a6040a6939bb54cf072234a (127.3.127.131). New cstate: current_term: 1 leader_uuid: "b85b0ea54a6040a6939bb54cf072234a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "050d9b44f01e4abc93dfb133af08e3dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b6c74b73eeb412887a9e933f11212c9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b85b0ea54a6040a6939bb54cf072234a" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42505 } health_report { overall_health: HEALTHY } } }
W20260812 02:41:31.691929 2811 tablet.cc:2369] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:41:31.737286 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:31.733468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51234 (local address 127.3.127.131:42505)
0812 02:41:31.733623 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:31.733628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:31.733644 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:41:31.733790 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:31.733793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:31.733857 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:31.733956 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:41:31.734230 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.734750 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.735658 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:31.735879 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:31.735971 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:31.736327 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:41:31.736767 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:41:31.736912 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:31.737043 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 02:41:31.737088 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260812 02:41:31.741334 2581 tablet.cc:2369] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:41:31.743618 2279 raft_consensus.cc:1261] T 561f70a5ae04495c989740f6167ec8e4 P 050d9b44f01e4abc93dfb133af08e3dd [term 1 FOLLOWER]: Refusing update from remote peer b85b0ea54a6040a6939bb54cf072234a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:41:31.743618 2506 raft_consensus.cc:1261] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9 [term 1 FOLLOWER]: Refusing update from remote peer b85b0ea54a6040a6939bb54cf072234a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:41:31.744415 2855 consensus_queue.cc:1035] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b6c74b73eeb412887a9e933f11212c9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:41:31.744625 2853 consensus_queue.cc:1035] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a [LEADER]: Connected to new peer: Peer: permanent_uuid: "050d9b44f01e4abc93dfb133af08e3dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 43637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:41:31.753237 2862 mvcc.cc:205] Tried to move back new op lower bound from 7317514206173999104 to 7317514205953318912. Current Snapshot: MvccSnapshot[applied={T|T < 7317514206173999104}]
I20260812 02:41:31.755062 2863 mvcc.cc:205] Tried to move back new op lower bound from 7317514206173999104 to 7317514205953318912. Current Snapshot: MvccSnapshot[applied={T|T < 7317514206173999104}]
I20260812 02:41:31.766921 2040 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:41:31.767131 2040 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:41:31.770078 2040 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db211b68b67249738cbf959241b929a4: Sending DeleteTablet for 3 replicas of tablet 561f70a5ae04495c989740f6167ec8e4
I20260812 02:41:31.771109 2246 tablet_service.cc:1514] Processing DeleteTablet for tablet 561f70a5ae04495c989740f6167ec8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:41:31 UTC) from {username='slave'} at 127.0.0.1:60284
I20260812 02:41:31.771275 2470 tablet_service.cc:1514] Processing DeleteTablet for tablet 561f70a5ae04495c989740f6167ec8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:41:31 UTC) from {username='slave'} at 127.0.0.1:47964
I20260812 02:41:31.771823 2878 tablet_replica.cc:330] T 561f70a5ae04495c989740f6167ec8e4 P 050d9b44f01e4abc93dfb133af08e3dd: stopping tablet replica
I20260812 02:41:31.772419 2696 tablet_service.cc:1514] Processing DeleteTablet for tablet 561f70a5ae04495c989740f6167ec8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:41:31 UTC) from {username='slave'} at 127.0.0.1:51224
I20260812 02:41:31.772653 2878 raft_consensus.cc:2229] T 561f70a5ae04495c989740f6167ec8e4 P 050d9b44f01e4abc93dfb133af08e3dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:41:31.773026 2878 raft_consensus.cc:2258] T 561f70a5ae04495c989740f6167ec8e4 P 050d9b44f01e4abc93dfb133af08e3dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:41:31.773305 2880 tablet_replica.cc:330] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9: stopping tablet replica
I20260812 02:41:31.773370 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 2136
I20260812 02:41:31.781992 2883 tablet_replica.cc:330] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a: stopping tablet replica
I20260812 02:41:31.782546 2883 raft_consensus.cc:2229] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:41:31.783196 2883 raft_consensus.cc:2258] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:41:31.792876 2883 ts_tablet_manager.cc:1905] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:41:31.796694 2883 ts_tablet_manager.cc:1918] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:41:31.796852 2883 log.cc:1196] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/561f70a5ae04495c989740f6167ec8e4
I20260812 02:41:31.797307 2883 ts_tablet_manager.cc:1939] T 561f70a5ae04495c989740f6167ec8e4 P b85b0ea54a6040a6939bb54cf072234a: Deleting consensus metadata
I20260812 02:41:31.799173 2016 catalog_manager.cc:4841] TS b85b0ea54a6040a6939bb54cf072234a (127.3.127.131:42505): tablet 561f70a5ae04495c989740f6167ec8e4 (table test-table [id=88663976e0494795af91c1654af04f03]) successfully deleted
I20260812 02:41:31.806284 2880 raft_consensus.cc:2229] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:41:31.806751 2880 raft_consensus.cc:2258] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:41:31.808555 2880 ts_tablet_manager.cc:1905] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:41:31.812358 2880 ts_tablet_manager.cc:1918] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:41:31.812542 2880 log.cc:1196] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/561f70a5ae04495c989740f6167ec8e4
I20260812 02:41:31.812909 2880 ts_tablet_manager.cc:1939] T 561f70a5ae04495c989740f6167ec8e4 P 5b6c74b73eeb412887a9e933f11212c9: Deleting consensus metadata
I20260812 02:41:31.812917 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 2370
W20260812 02:41:31.812891 2016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.127.129:43637 (error 108)
I20260812 02:41:31.826483 2826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:31.826173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:43637 (local address 127.0.0.1:60290)
0812 02:41:31.826307 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:31.826388 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.129:43637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 02:41:31.826768 2016 catalog_manager.cc:4584] TS 050d9b44f01e4abc93dfb133af08e3dd (127.3.127.129:43637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 561f70a5ae04495c989740f6167ec8e4: Network error: Client connection negotiation failed: client connection to 127.3.127.129:43637: connect: Connection refused (error 111)
I20260812 02:41:31.837946 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 2583
I20260812 02:41:31.838052 2826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:31.836322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:38173 (local address 127.0.0.1:47972)
0812 02:41:31.837869 (+ 1547us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:31.837940 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.130:38173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1426}
I20260812 02:41:31.857214 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 1994
2026-08-12T02:41:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1343 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:41:31 dist-test-slave-rmhg krb5kdc[2894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:41:31 dist-test-slave-rmhg krb5kdc[2894](info): set up 2 sockets
Aug 12 02:41:31 dist-test-slave-rmhg krb5kdc[2894](info): commencing operation
krb5kdc: starting...
W20260812 02:41:33.045696 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:41:33 dist-test-slave-rmhg krb5kdc[2894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:41:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:41:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:41:33.180816 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:41403
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:41403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:33.298836 2917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:33.299180 2917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:33.299245 2917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:33.303470 2917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:33.303561 2917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:33.303591 2917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:33.303613 2917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:33.303633 2917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:33.309401 2917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:41403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:41403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:33.310657 2917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:33.311764 2917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:33.320735 2923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:33.320868 2917 server_base.cc:1034] running on GCE node
W20260812 02:41:33.320735 2927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:33.320783 2922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:33.321691 2917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:33.322896 2917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:33.324071 2917 hybrid_clock.cc:648] HybridClock initialized: now 1786502493324052 us; error 37 us; skew 500 ppm
Aug 12 02:41:33 dist-test-slave-rmhg krb5kdc[2894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502493, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:33.327070 2917 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:41:33.328083 2917 webserver.cc:460] Webserver started at http://127.3.127.190:42085/ using document root <none> and password file <none>
I20260812 02:41:33.328660 2917 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:33.328727 2917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:33.328913 2917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:33.331194 2917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "c80a395d4c1d4929965649caa88e978a"
format_stamp: "Formatted at 2026-08-12 02:41:33 on dist-test-slave-rmhg"
server_key: "7ce18b6f3b1132e399661297a111ef9a"
server_key_iv: "945d09db6f4e162a07b12486210c10b9"
server_key_version: "encryptionkey@0"
I20260812 02:41:33.331810 2917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "c80a395d4c1d4929965649caa88e978a"
format_stamp: "Formatted at 2026-08-12 02:41:33 on dist-test-slave-rmhg"
server_key: "7ce18b6f3b1132e399661297a111ef9a"
server_key_iv: "945d09db6f4e162a07b12486210c10b9"
server_key_version: "encryptionkey@0"
I20260812 02:41:33.336139 2917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:41:33.339134 2939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:33.340714 2917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:41:33.340880 2917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "c80a395d4c1d4929965649caa88e978a"
format_stamp: "Formatted at 2026-08-12 02:41:33 on dist-test-slave-rmhg"
server_key: "7ce18b6f3b1132e399661297a111ef9a"
server_key_iv: "945d09db6f4e162a07b12486210c10b9"
server_key_version: "encryptionkey@0"
I20260812 02:41:33.341355 2917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:33.374671 2917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:33.378281 2917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:33.378484 2917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:33.386118 2917 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:41403
I20260812 02:41:33.386183 3018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:41403 every 8 connection(s)
I20260812 02:41:33.387166 2917 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:33.389770 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 2917
I20260812 02:41:33.389920 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:41:33.390275 3582 external_mini_cluster.cc:1428] Setting key 56cba145113b18c9b34c38bd8b3bc5b0
I20260812 02:41:33.390460 3020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:33.397886 3020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a: Bootstrap starting.
Aug 12 02:41:33 dist-test-slave-rmhg krb5kdc[2894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:41:33.400736 3020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:33.401546 3020 log.cc:824] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:33.403824 3020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a: No bootstrap required, opened a new log
I20260812 02:41:33.405717 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:33.391741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.3.127.190:41403)
0812 02:41:33.392211 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:33.392228 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:33.392266 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:41:33.393296 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:33.393319 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:33.393702 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:33.394004 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:33.394993 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:33.396020 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:33.397242 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:33.397724 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:33.397851 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:33.400627 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:33.400633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:33.400649 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:33.400678 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:33.402515 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:33.403035 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:33.403039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:33.403044 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:33.403172 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:33.403447 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:33.403449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:33.403450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:33.403852 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:33.404037 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:33.404348 (+ 311us) server_negotiation.cc:299] Negotiation successful
0812 02:41:33.404670 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":145,"threads_started":1}
I20260812 02:41:33.407207 3020 raft_consensus.cc:348] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80a395d4c1d4929965649caa88e978a" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41403 } }
I20260812 02:41:33.407472 3020 raft_consensus.cc:374] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:33.407543 3020 raft_consensus.cc:729] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c80a395d4c1d4929965649caa88e978a, State: Initialized, Role: FOLLOWER
I20260812 02:41:33.407998 3020 consensus_queue.cc:260] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80a395d4c1d4929965649caa88e978a" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41403 } }
I20260812 02:41:33.408179 3020 raft_consensus.cc:388] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:33.408236 3020 raft_consensus.cc:482] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:33.408315 3020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:33.409284 3020 raft_consensus.cc:504] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80a395d4c1d4929965649caa88e978a" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41403 } }
I20260812 02:41:33.409608 3020 leader_election.cc:302] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c80a395d4c1d4929965649caa88e978a; no voters:
I20260812 02:41:33.409909 3020 leader_election.cc:288] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:41:33.410068 3026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:33.410295 3026 raft_consensus.cc:686] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a [term 1 LEADER]: Becoming Leader. State: Replica: c80a395d4c1d4929965649caa88e978a, State: Running, Role: LEADER
I20260812 02:41:33.410694 3026 consensus_queue.cc:237] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80a395d4c1d4929965649caa88e978a" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41403 } }
I20260812 02:41:33.411150 3020 sys_catalog.cc:564] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:33.412230 3027 sys_catalog.cc:455] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c80a395d4c1d4929965649caa88e978a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80a395d4c1d4929965649caa88e978a" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41403 } } }
I20260812 02:41:33.412367 3027 sys_catalog.cc:458] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:33.412452 3028 sys_catalog.cc:455] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c80a395d4c1d4929965649caa88e978a. Latest consensus state: current_term: 1 leader_uuid: "c80a395d4c1d4929965649caa88e978a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80a395d4c1d4929965649caa88e978a" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 41403 } } }
I20260812 02:41:33.412534 3028 sys_catalog.cc:458] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:33.412870 3042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:33.415760 3042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:33.422051 3042 catalog_manager.cc:1347] Generated new cluster ID: e25829972926453985865517ee848a47
I20260812 02:41:33.422148 3042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:33.434175 3042 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:41:33.435576 3042 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:33.448426 3042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a: Generated new TSK 0
I20260812 02:41:33.449345 3042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
Entry for principal kudu/127.3.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab.
I20260812 02:41:33.488540 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:41403
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:33.611833 3058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:33.612176 3058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:33.612231 3058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:33.616416 3058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:33.616489 3058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:33.616550 3058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:41:33.622068 3058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.129.keytab
--principal=kudu/127.3.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:41403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:33.623214 3058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:33.624369 3058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:33.634577 3080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:33.634577 3073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:33.634580 3071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:33.634997 3058 server_base.cc:1034] running on GCE node
I20260812 02:41:33.635506 3058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:33.636272 3058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:33.637439 3058 hybrid_clock.cc:648] HybridClock initialized: now 1786502493637419 us; error 42 us; skew 500 ppm
Aug 12 02:41:33 dist-test-slave-rmhg krb5kdc[2894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502493, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:33.640357 3058 init.cc:377] Logged in from keytab as kudu/127.3.127.129@KRBTEST.COM (short username kudu)
I20260812 02:41:33.641425 3058 webserver.cc:460] Webserver started at http://127.3.127.129:41909/ using document root <none> and password file <none>
I20260812 02:41:33.642019 3058 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:33.642091 3058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:33.642314 3058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:33.644479 3058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "cc853a7e128a4d21b54e48ae80f8fb2c"
format_stamp: "Formatted at 2026-08-12 02:41:33 on dist-test-slave-rmhg"
server_key: "025c426be8d01f3e6053380bd03d68c5"
server_key_iv: "15a2e061a8a5692c88b274260521d840"
server_key_version: "encryptionkey@0"
I20260812 02:41:33.645051 3058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "cc853a7e128a4d21b54e48ae80f8fb2c"
format_stamp: "Formatted at 2026-08-12 02:41:33 on dist-test-slave-rmhg"
server_key: "025c426be8d01f3e6053380bd03d68c5"
server_key_iv: "15a2e061a8a5692c88b274260521d840"
server_key_version: "encryptionkey@0"
I20260812 02:41:33.649245 3058 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:41:33.652027 3092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:33.653401 3058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:41:33.653520 3058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "cc853a7e128a4d21b54e48ae80f8fb2c"
format_stamp: "Formatted at 2026-08-12 02:41:33 on dist-test-slave-rmhg"
server_key: "025c426be8d01f3e6053380bd03d68c5"
server_key_iv: "15a2e061a8a5692c88b274260521d840"
server_key_version: "encryptionkey@0"
I20260812 02:41:33.653628 3058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:33.669049 3058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:33.672430 3058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:33.672631 3058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:33.673297 3058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:41:33.674175 3058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:41:33.674235 3058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:33.674290 3058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:41:33.674315 3058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:33.683185 3058 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:42063
I20260812 02:41:33.683215 3271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:42063 every 8 connection(s)
I20260812 02:41:33.684092 3058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:41:33.686880 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 3058
I20260812 02:41:33.687009 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:41:33.687346 3582 external_mini_cluster.cc:1428] Setting key 28766841c2fa35144a791221fa1742ef
Aug 12 02:41:33 dist-test-slave-rmhg krb5kdc[2894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502493, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.129@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.127.130@KRBTEST.COM; defaulting to no policy
I20260812 02:41:33.700039 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:33.686008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:57559 (local address 127.3.127.190:41403)
0812 02:41:33.686210 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:33.686216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:33.687069 (+ 853us) server_negotiation.cc:407] Connection header received
0812 02:41:33.688614 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:33.688618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:33.688668 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:33.688761 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:33.690824 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:33.691677 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:33.692880 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:33.693234 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:33.693343 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:33.696223 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:33.696226 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:33.696228 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:33.696255 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:33.697743 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:33.698357 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:33.698360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:33.698361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:33.698445 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:33.698843 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:33.698846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:33.698847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:33.699069 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:33.699157 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:33.699763 (+ 606us) server_negotiation.cc:299] Negotiation successful
0812 02:41:33.699891 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.130@KRBTEST.COM" created.
I20260812 02:41:33.701114 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:33.686415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41403 (local address 127.3.127.129:57559)
0812 02:41:33.686906 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:33.686959 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:33.688363 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:33.689445 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:33.689456 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:33.689957 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:41:33.690649 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:33.690669 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:33.691801 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:33.691804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:33.692747 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:33.692757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:33.693375 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:33.693378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:33.693567 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:33.694191 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:41:33.694208 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:41:33.696037 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:41:33.697875 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:33.697882 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:33.697902 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:33.698257 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:33.698547 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:33.698551 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:33.698552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:33.698725 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:33.699202 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:41:33.699208 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:41:33.699475 (+ 267us) client_negotiation.cc:769] Sending connection context
0812 02:41:33.699699 (+ 224us) client_negotiation.cc:240] Negotiation successful
0812 02:41:33.699991 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":120,"threads_started":1}
I20260812 02:41:33.702502 3273 heartbeater.cc:347] Connected to a master server at 127.3.127.190:41403
I20260812 02:41:33.702775 3273 heartbeater.cc:464] Registering TS with master...
I20260812 02:41:33.703333 3273 heartbeater.cc:511] Master 127.3.127.190:41403 requested a full tablet report, sending...
I20260812 02:41:33.705061 2964 ts_manager.cc:194] Registered new tserver with Master: cc853a7e128a4d21b54e48ae80f8fb2c (127.3.127.129:42063)
I20260812 02:41:33.706836 2964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.129@KRBTEST.COM'} at 127.3.127.129:57559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
Entry for principal kudu/127.3.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab.
I20260812 02:41:33.718189 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:41403
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:33.836644 3284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:33.836948 3284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:33.836994 3284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:33.841230 3284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:33.841305 3284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:33.841367 3284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:41:33.847378 3284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.130.keytab
--principal=kudu/127.3.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:41403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:33.848529 3284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:33.849644 3284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:33.860064 3291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:33.860193 3284 server_base.cc:1034] running on GCE node
W20260812 02:41:33.860049 3292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:33.860049 3296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:33.860749 3284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:33.861481 3284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:33.862646 3284 hybrid_clock.cc:648] HybridClock initialized: now 1786502493862624 us; error 37 us; skew 500 ppm
Aug 12 02:41:33 dist-test-slave-rmhg krb5kdc[2894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502493, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:33.865490 3284 init.cc:377] Logged in from keytab as kudu/127.3.127.130@KRBTEST.COM (short username kudu)
I20260812 02:41:33.866429 3284 webserver.cc:460] Webserver started at http://127.3.127.130:38871/ using document root <none> and password file <none>
I20260812 02:41:33.866973 3284 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:33.867031 3284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:33.867213 3284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:33.869391 3284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "c6f76c00992b44d193eb8ca26049a579"
format_stamp: "Formatted at 2026-08-12 02:41:33 on dist-test-slave-rmhg"
server_key: "a80f6ccb6b7aa080c7d1822af772a331"
server_key_iv: "0c5cbc6570b44469c424a919a48b54b3"
server_key_version: "encryptionkey@0"
I20260812 02:41:33.869930 3284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "c6f76c00992b44d193eb8ca26049a579"
format_stamp: "Formatted at 2026-08-12 02:41:33 on dist-test-slave-rmhg"
server_key: "a80f6ccb6b7aa080c7d1822af772a331"
server_key_iv: "0c5cbc6570b44469c424a919a48b54b3"
server_key_version: "encryptionkey@0"
I20260812 02:41:33.873823 3284 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:41:33.876442 3311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:33.877767 3284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:41:33.877890 3284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "c6f76c00992b44d193eb8ca26049a579"
format_stamp: "Formatted at 2026-08-12 02:41:33 on dist-test-slave-rmhg"
server_key: "a80f6ccb6b7aa080c7d1822af772a331"
server_key_iv: "0c5cbc6570b44469c424a919a48b54b3"
server_key_version: "encryptionkey@0"
I20260812 02:41:33.878010 3284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:33.897504 3284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:33.900899 3284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:33.901098 3284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:33.901854 3284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:41:33.902920 3284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:41:33.902978 3284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:33.903031 3284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:41:33.903062 3284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:33.912339 3284 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:42227
I20260812 02:41:33.912376 3520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:42227 every 8 connection(s)
I20260812 02:41:33.913439 3284 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:41:33.916071 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 3284
I20260812 02:41:33.916211 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:41:33.916540 3582 external_mini_cluster.cc:1428] Setting key 822546e141508aaaedfba800dd58891b
Aug 12 02:41:33 dist-test-slave-rmhg krb5kdc[2894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502493, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.130@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.127.131@KRBTEST.COM; defaulting to no policy
I20260812 02:41:33.928196 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:33.915386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:36395 (local address 127.3.127.190:41403)
0812 02:41:33.915510 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:33.915514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:33.916517 (+ 1003us) server_negotiation.cc:407] Connection header received
0812 02:41:33.918120 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:33.918123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:33.918179 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:33.918249 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:33.919837 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:33.920385 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:33.921674 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:33.921897 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:33.921984 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:33.924604 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:33.924607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:33.924609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:33.924635 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:33.926003 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:33.926635 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:33.926638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:33.926639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:33.926715 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:33.927068 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:33.927070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:33.927071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:33.927268 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:33.927344 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:33.927909 (+ 565us) server_negotiation.cc:299] Negotiation successful
0812 02:41:33.928035 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 02:41:33.928948 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:33.915784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41403 (local address 127.3.127.130:36395)
0812 02:41:33.916325 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:33.916391 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:33.917898 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:33.918439 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:33.918449 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:33.918965 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:41:33.919679 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:33.919699 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:33.920528 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:33.920532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:33.921540 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:33.921551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:33.922049 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:33.922053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:33.922234 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:33.922800 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:41:33.922817 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:41:33.924407 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:41:33.926134 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:33.926141 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:33.926159 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:33.926514 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:33.926802 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:33.926805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:33.926807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:33.926986 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:33.927370 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:41:33.927377 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:41:33.927643 (+ 266us) client_negotiation.cc:769] Sending connection context
0812 02:41:33.927811 (+ 168us) client_negotiation.cc:240] Negotiation successful
0812 02:41:33.928091 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.131@KRBTEST.COM" created.
I20260812 02:41:33.930387 3521 heartbeater.cc:347] Connected to a master server at 127.3.127.190:41403
I20260812 02:41:33.930639 3521 heartbeater.cc:464] Registering TS with master...
I20260812 02:41:33.931162 3521 heartbeater.cc:511] Master 127.3.127.190:41403 requested a full tablet report, sending...
I20260812 02:41:33.932219 2964 ts_manager.cc:194] Registered new tserver with Master: c6f76c00992b44d193eb8ca26049a579 (127.3.127.130:42227)
I20260812 02:41:33.932801 2964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.130@KRBTEST.COM'} at 127.3.127.130:36395
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
Entry for principal kudu/127.3.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab.
I20260812 02:41:33.946694 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:41403
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:34.066761 3530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:34.067094 3530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:34.067143 3530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:34.071452 3530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:34.071537 3530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:34.071601 3530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:41:34.077457 3530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.131.keytab
--principal=kudu/127.3.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:41403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:34.078673 3530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:34.079800 3530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:41:34.090570 3530 server_base.cc:1034] running on GCE node
W20260812 02:41:34.090521 3545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:34.090523 3536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:34.090520 3537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:34.091327 3530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:34.092087 3530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:34.093228 3530 hybrid_clock.cc:648] HybridClock initialized: now 1786502494093205 us; error 44 us; skew 500 ppm
Aug 12 02:41:34 dist-test-slave-rmhg krb5kdc[2894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502494, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:34.096288 3530 init.cc:377] Logged in from keytab as kudu/127.3.127.131@KRBTEST.COM (short username kudu)
I20260812 02:41:34.097388 3530 webserver.cc:460] Webserver started at http://127.3.127.131:41917/ using document root <none> and password file <none>
I20260812 02:41:34.098001 3530 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:34.098078 3530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:34.098311 3530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:34.100548 3530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "a1c8691b056b47b1b220ece9348bfaaf"
format_stamp: "Formatted at 2026-08-12 02:41:34 on dist-test-slave-rmhg"
server_key: "a36e52661cef1860ed8d0f5dab854648"
server_key_iv: "a58d02f93ede9d1837fb893cd8f6d2de"
server_key_version: "encryptionkey@0"
I20260812 02:41:34.101210 3530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "a1c8691b056b47b1b220ece9348bfaaf"
format_stamp: "Formatted at 2026-08-12 02:41:34 on dist-test-slave-rmhg"
server_key: "a36e52661cef1860ed8d0f5dab854648"
server_key_iv: "a58d02f93ede9d1837fb893cd8f6d2de"
server_key_version: "encryptionkey@0"
I20260812 02:41:34.105357 3530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260812 02:41:34.108129 3555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:34.109544 3530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:41:34.109704 3530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "a1c8691b056b47b1b220ece9348bfaaf"
format_stamp: "Formatted at 2026-08-12 02:41:34 on dist-test-slave-rmhg"
server_key: "a36e52661cef1860ed8d0f5dab854648"
server_key_iv: "a58d02f93ede9d1837fb893cd8f6d2de"
server_key_version: "encryptionkey@0"
I20260812 02:41:34.109820 3530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:34.130755 3530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:34.134079 3530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:34.134281 3530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:34.134984 3530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:41:34.135923 3530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:41:34.135977 3530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:34.136034 3530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:41:34.136061 3530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:34.145326 3530 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:42587
I20260812 02:41:34.145387 3760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:42587 every 8 connection(s)
I20260812 02:41:34.146330 3530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:41:34.155285 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 3530
I20260812 02:41:34.155414 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:41:34.155696 3582 external_mini_cluster.cc:1428] Setting key 8944784c36c5324ac7a7257781af6c62
Aug 12 02:41:34 dist-test-slave-rmhg krb5kdc[2894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502494, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.131@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:41:34.160836 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:34.148254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:43535 (local address 127.3.127.190:41403)
0812 02:41:34.148405 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:34.148409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:34.149404 (+ 995us) server_negotiation.cc:407] Connection header received
0812 02:41:34.150840 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:34.150844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:34.150907 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:34.150997 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:34.152324 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.152866 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.154123 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.154354 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.154446 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:34.157313 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:34.157317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:34.157320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:34.157348 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:34.158855 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:34.159352 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:34.159355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:34.159356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:34.159436 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:34.159794 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:34.159798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:34.159799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:34.160031 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:34.160115 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:34.160596 (+ 481us) server_negotiation.cc:299] Negotiation successful
0812 02:41:34.160712 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 02:41:34.161789 3763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:34.148662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:41403 (local address 127.3.127.131:43535)
0812 02:41:34.149219 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:34.149268 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:34.150600 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:34.151189 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:34.151197 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:34.151631 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:41:34.152163 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:34.152179 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.153000 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.153003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:34.153997 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:34.154008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.154517 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.154521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:34.154701 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:34.155305 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:41:34.155323 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:41:34.157113 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:41:34.158963 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:34.158968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:34.158982 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:34.159261 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:34.159506 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:41:34.159508 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:41:34.159509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:41:34.159667 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:41:34.160115 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:41:34.160120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:41:34.160382 (+ 262us) client_negotiation.cc:769] Sending connection context
0812 02:41:34.160538 (+ 156us) client_negotiation.cc:240] Negotiation successful
0812 02:41:34.160847 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":124,"threads_started":1}
I20260812 02:41:34.162927 3761 heartbeater.cc:347] Connected to a master server at 127.3.127.190:41403
I20260812 02:41:34.163213 3761 heartbeater.cc:464] Registering TS with master...
I20260812 02:41:34.163755 3761 heartbeater.cc:511] Master 127.3.127.190:41403 requested a full tablet report, sending...
I20260812 02:41:34.164849 2964 ts_manager.cc:194] Registered new tserver with Master: a1c8691b056b47b1b220ece9348bfaaf (127.3.127.131:42587)
I20260812 02:41:34.165621 2964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.127.131@KRBTEST.COM'} at 127.3.127.131:43535
I20260812 02:41:34.170102 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:41:34.179198 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:34.172154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35308 (local address 127.3.127.190:41403)
0812 02:41:34.172282 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:34.172286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:34.172445 (+ 159us) server_negotiation.cc:407] Connection header received
0812 02:41:34.172559 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:34.172562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:34.172604 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:34.172672 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:34.173402 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.173955 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.174827 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.175048 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.175125 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:34.176484 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:34.176488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:34.176490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:34.176513 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:34.177774 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:34.178171 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:34.178173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:34.178174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:34.178248 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:34.178437 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:34.178439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:34.178440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:34.178666 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:34.178770 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:34.178945 (+ 175us) server_negotiation.cc:299] Negotiation successful
0812 02:41:34.179054 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 02:41:34.182292 2964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:41:34.185422 2964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:41:34.200151 3783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:34.194511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:42587 (local address 127.0.0.1:34286)
0812 02:41:34.195457 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:34.195470 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:34.195563 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:34.196152 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:34.196155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:34.196173 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:34.196277 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:34.196284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.197367 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.197370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:34.198541 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:34.198550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.199607 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.199611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:34.199718 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:34.199747 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 02:41:34.199822 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 02:41:34.199998 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"mutex_wait_us":4,"thread_start_us":131,"threads_started":1}
I20260812 02:41:34.200152 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:34.194312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47518 (local address 127.3.127.129:42063)
0812 02:41:34.194759 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:34.194781 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:34.194833 (+ 52us) server_negotiation.cc:407] Connection header received
0812 02:41:34.194918 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:34.194924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:34.195166 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:34.195334 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:34.195880 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.197072 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.198777 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.199447 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.199554 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:34.199748 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:34.199869 (+ 121us) server_negotiation.cc:299] Negotiation successful
0812 02:41:34.199973 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":94,"threads_started":1}
I20260812 02:41:34.200340 3782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:34.194908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:42227 (local address 127.0.0.1:60744)
0812 02:41:34.195751 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:34.195763 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:34.195904 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:34.196417 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:34.196419 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:34.196438 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:34.196517 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:34.196523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.197831 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.197834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:34.198929 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:34.198938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.199955 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.199958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:34.200042 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:34.200064 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 02:41:34.200146 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 02:41:34.200213 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":85,"threads_started":1}
I20260812 02:41:34.200416 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:34.195071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60744 (local address 127.3.127.130:42227)
0812 02:41:34.195484 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:34.195501 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:34.195841 (+ 340us) server_negotiation.cc:407] Connection header received
0812 02:41:34.196087 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:34.196093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:34.196292 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:34.196441 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:34.196595 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.197680 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.199060 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.199860 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.199951 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:34.200084 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:34.200182 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 02:41:34.200274 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":134,"threads_started":1}
I20260812 02:41:34.200994 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:34.195008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34286 (local address 127.3.127.131:42587)
0812 02:41:34.195694 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:34.195705 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:34.195729 (+ 24us) server_negotiation.cc:407] Connection header received
0812 02:41:34.195809 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:34.195815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:34.196021 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:34.196174 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:34.196383 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.197226 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.198687 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.199462 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.200577 (+ 1115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:34.200723 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:34.200765 (+ 42us) server_negotiation.cc:299] Negotiation successful
0812 02:41:34.200858 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":144,"threads_started":1}
I20260812 02:41:34.201344 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:34.194116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:42063 (local address 127.0.0.1:47518)
0812 02:41:34.194511 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:34.194576 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:34.194731 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:34.195557 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:34.195564 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:34.195587 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:34.195744 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:34.195752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.197261 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.197265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:34.198630 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:34.198641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.199574 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.199577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:34.199690 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:34.199729 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 02:41:34.199839 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 02:41:34.201218 (+ 1379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":1308,"thread_start_us":102,"threads_started":1}
I20260812 02:41:34.203871 3649 tablet_service.cc:1467] Processing CreateTablet for tablet 2e100c58d8574272b0ff4a08683ca4f4 (DEFAULT_TABLE table=test-table [id=ee5c593187914de19ff703375464fa86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:41:34.203866 3391 tablet_service.cc:1467] Processing CreateTablet for tablet 2e100c58d8574272b0ff4a08683ca4f4 (DEFAULT_TABLE table=test-table [id=ee5c593187914de19ff703375464fa86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:41:34.204200 3170 tablet_service.cc:1467] Processing CreateTablet for tablet 2e100c58d8574272b0ff4a08683ca4f4 (DEFAULT_TABLE table=test-table [id=ee5c593187914de19ff703375464fa86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:41:34.204933 3649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e100c58d8574272b0ff4a08683ca4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:34.204990 3391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e100c58d8574272b0ff4a08683ca4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:34.205107 3170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e100c58d8574272b0ff4a08683ca4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:34.211817 3786 tablet_bootstrap.cc:492] T 2e100c58d8574272b0ff4a08683ca4f4 P a1c8691b056b47b1b220ece9348bfaaf: Bootstrap starting.
I20260812 02:41:34.212958 3787 tablet_bootstrap.cc:492] T 2e100c58d8574272b0ff4a08683ca4f4 P cc853a7e128a4d21b54e48ae80f8fb2c: Bootstrap starting.
I20260812 02:41:34.213503 3789 tablet_bootstrap.cc:492] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579: Bootstrap starting.
I20260812 02:41:34.214591 3786 tablet_bootstrap.cc:654] T 2e100c58d8574272b0ff4a08683ca4f4 P a1c8691b056b47b1b220ece9348bfaaf: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:34.215443 3787 tablet_bootstrap.cc:654] T 2e100c58d8574272b0ff4a08683ca4f4 P cc853a7e128a4d21b54e48ae80f8fb2c: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:34.215842 3789 tablet_bootstrap.cc:654] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:34.215878 3786 log.cc:824] T 2e100c58d8574272b0ff4a08683ca4f4 P a1c8691b056b47b1b220ece9348bfaaf: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:34.216399 3787 log.cc:824] T 2e100c58d8574272b0ff4a08683ca4f4 P cc853a7e128a4d21b54e48ae80f8fb2c: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:34.217213 3789 log.cc:824] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:34.218454 3786 tablet_bootstrap.cc:492] T 2e100c58d8574272b0ff4a08683ca4f4 P a1c8691b056b47b1b220ece9348bfaaf: No bootstrap required, opened a new log
I20260812 02:41:34.218677 3786 ts_tablet_manager.cc:1397] T 2e100c58d8574272b0ff4a08683ca4f4 P a1c8691b056b47b1b220ece9348bfaaf: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260812 02:41:34.218747 3787 tablet_bootstrap.cc:492] T 2e100c58d8574272b0ff4a08683ca4f4 P cc853a7e128a4d21b54e48ae80f8fb2c: No bootstrap required, opened a new log
I20260812 02:41:34.218967 3787 ts_tablet_manager.cc:1397] T 2e100c58d8574272b0ff4a08683ca4f4 P cc853a7e128a4d21b54e48ae80f8fb2c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 02:41:34.219579 3789 tablet_bootstrap.cc:492] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579: No bootstrap required, opened a new log
I20260812 02:41:34.219789 3789 ts_tablet_manager.cc:1397] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 02:41:34.222218 3786 raft_consensus.cc:348] T 2e100c58d8574272b0ff4a08683ca4f4 P a1c8691b056b47b1b220ece9348bfaaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc853a7e128a4d21b54e48ae80f8fb2c" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 42063 } } peers { permanent_uuid: "c6f76c00992b44d193eb8ca26049a579" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 42227 } } peers { permanent_uuid: "a1c8691b056b47b1b220ece9348bfaaf" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42587 } }
I20260812 02:41:34.222519 3786 raft_consensus.cc:374] T 2e100c58d8574272b0ff4a08683ca4f4 P a1c8691b056b47b1b220ece9348bfaaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:34.222600 3786 raft_consensus.cc:729] T 2e100c58d8574272b0ff4a08683ca4f4 P a1c8691b056b47b1b220ece9348bfaaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1c8691b056b47b1b220ece9348bfaaf, State: Initialized, Role: FOLLOWER
I20260812 02:41:34.223098 3786 consensus_queue.cc:260] T 2e100c58d8574272b0ff4a08683ca4f4 P a1c8691b056b47b1b220ece9348bfaaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc853a7e128a4d21b54e48ae80f8fb2c" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 42063 } } peers { permanent_uuid: "c6f76c00992b44d193eb8ca26049a579" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 42227 } } peers { permanent_uuid: "a1c8691b056b47b1b220ece9348bfaaf" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42587 } }
I20260812 02:41:34.223775 3787 raft_consensus.cc:348] T 2e100c58d8574272b0ff4a08683ca4f4 P cc853a7e128a4d21b54e48ae80f8fb2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc853a7e128a4d21b54e48ae80f8fb2c" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 42063 } } peers { permanent_uuid: "c6f76c00992b44d193eb8ca26049a579" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 42227 } } peers { permanent_uuid: "a1c8691b056b47b1b220ece9348bfaaf" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42587 } }
I20260812 02:41:34.223775 3789 raft_consensus.cc:348] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc853a7e128a4d21b54e48ae80f8fb2c" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 42063 } } peers { permanent_uuid: "c6f76c00992b44d193eb8ca26049a579" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 42227 } } peers { permanent_uuid: "a1c8691b056b47b1b220ece9348bfaaf" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42587 } }
I20260812 02:41:34.224077 3789 raft_consensus.cc:374] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:34.224149 3787 raft_consensus.cc:374] T 2e100c58d8574272b0ff4a08683ca4f4 P cc853a7e128a4d21b54e48ae80f8fb2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:34.224205 3789 raft_consensus.cc:729] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f76c00992b44d193eb8ca26049a579, State: Initialized, Role: FOLLOWER
I20260812 02:41:34.224246 3787 raft_consensus.cc:729] T 2e100c58d8574272b0ff4a08683ca4f4 P cc853a7e128a4d21b54e48ae80f8fb2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc853a7e128a4d21b54e48ae80f8fb2c, State: Initialized, Role: FOLLOWER
I20260812 02:41:34.224326 3786 ts_tablet_manager.cc:1428] T 2e100c58d8574272b0ff4a08683ca4f4 P a1c8691b056b47b1b220ece9348bfaaf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 02:41:34.224831 3761 heartbeater.cc:503] Master 127.3.127.190:41403 was elected leader, sending a full tablet report...
I20260812 02:41:34.224838 3789 consensus_queue.cc:260] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc853a7e128a4d21b54e48ae80f8fb2c" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 42063 } } peers { permanent_uuid: "c6f76c00992b44d193eb8ca26049a579" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 42227 } } peers { permanent_uuid: "a1c8691b056b47b1b220ece9348bfaaf" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42587 } }
I20260812 02:41:34.224838 3787 consensus_queue.cc:260] T 2e100c58d8574272b0ff4a08683ca4f4 P cc853a7e128a4d21b54e48ae80f8fb2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc853a7e128a4d21b54e48ae80f8fb2c" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 42063 } } peers { permanent_uuid: "c6f76c00992b44d193eb8ca26049a579" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 42227 } } peers { permanent_uuid: "a1c8691b056b47b1b220ece9348bfaaf" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42587 } }
I20260812 02:41:34.226028 3787 ts_tablet_manager.cc:1428] T 2e100c58d8574272b0ff4a08683ca4f4 P cc853a7e128a4d21b54e48ae80f8fb2c: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260812 02:41:34.226028 3789 ts_tablet_manager.cc:1428] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260812 02:41:34.226437 3273 heartbeater.cc:503] Master 127.3.127.190:41403 was elected leader, sending a full tablet report...
I20260812 02:41:34.226483 3521 heartbeater.cc:503] Master 127.3.127.190:41403 was elected leader, sending a full tablet report...
I20260812 02:41:34.282148 3798 raft_consensus.cc:482] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:41:34.282369 3798 raft_consensus.cc:504] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc853a7e128a4d21b54e48ae80f8fb2c" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 42063 } } peers { permanent_uuid: "c6f76c00992b44d193eb8ca26049a579" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 42227 } } peers { permanent_uuid: "a1c8691b056b47b1b220ece9348bfaaf" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42587 } }
I20260812 02:41:34.283627 3798 leader_election.cc:288] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc853a7e128a4d21b54e48ae80f8fb2c (127.3.127.129:42063), a1c8691b056b47b1b220ece9348bfaaf (127.3.127.131:42587)
I20260812 02:41:34.287761 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:34.283931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:42587 (local address 127.3.127.130:35897)
0812 02:41:34.284057 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:34.284073 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:34.284201 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:34.284482 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:34.284485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:34.284508 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:34.284637 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:34.284651 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.285568 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.285571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:34.286557 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:34.286568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.287313 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.287317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:34.287428 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:34.287455 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 02:41:34.287527 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 02:41:34.287609 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":1}
I20260812 02:41:34.287792 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:34.284032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:35897 (local address 127.3.127.131:42587)
0812 02:41:34.284179 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:34.284183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:34.284199 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:41:34.284308 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:34.284310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:34.284352 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:34.284423 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:34.284760 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.285450 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.286706 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.287186 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.287288 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:34.287333 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:34.287548 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 02:41:34.287624 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 02:41:34.288036 3799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:34.283933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:42063 (local address 127.3.127.130:48045)
0812 02:41:34.284676 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:34.284687 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:34.284786 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:34.285017 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:34.285020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:34.285038 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:34.285186 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:34.285193 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.285902 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.285905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:34.286788 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:34.286800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.287603 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.287606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:34.287710 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:34.287737 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 02:41:34.287819 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 02:41:34.287894 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":368,"thread_start_us":152,"threads_started":1}
I20260812 02:41:34.288053 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:34.284060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:48045 (local address 127.3.127.129:42063)
0812 02:41:34.284195 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:34.284200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:34.284754 (+ 554us) server_negotiation.cc:407] Connection header received
0812 02:41:34.284878 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:34.284880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:34.284923 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:34.284976 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:34.285272 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.285782 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.286927 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.287417 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.287605 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:34.287655 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:34.287848 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 02:41:34.287929 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 02:41:34.288726 3195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e100c58d8574272b0ff4a08683ca4f4" candidate_uuid: "c6f76c00992b44d193eb8ca26049a579" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc853a7e128a4d21b54e48ae80f8fb2c" is_pre_election: true
I20260812 02:41:34.288830 3682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e100c58d8574272b0ff4a08683ca4f4" candidate_uuid: "c6f76c00992b44d193eb8ca26049a579" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1c8691b056b47b1b220ece9348bfaaf" is_pre_election: true
I20260812 02:41:34.289075 3195 raft_consensus.cc:2454] T 2e100c58d8574272b0ff4a08683ca4f4 P cc853a7e128a4d21b54e48ae80f8fb2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6f76c00992b44d193eb8ca26049a579 in term 0.
I20260812 02:41:34.289126 3682 raft_consensus.cc:2454] T 2e100c58d8574272b0ff4a08683ca4f4 P a1c8691b056b47b1b220ece9348bfaaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6f76c00992b44d193eb8ca26049a579 in term 0.
I20260812 02:41:34.289630 3312 leader_election.cc:302] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6f76c00992b44d193eb8ca26049a579, cc853a7e128a4d21b54e48ae80f8fb2c; no voters:
I20260812 02:41:34.289913 3798 raft_consensus.cc:2781] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:41:34.290020 3798 raft_consensus.cc:482] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:41:34.290073 3798 raft_consensus.cc:3037] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:34.291182 3798 raft_consensus.cc:504] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc853a7e128a4d21b54e48ae80f8fb2c" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 42063 } } peers { permanent_uuid: "c6f76c00992b44d193eb8ca26049a579" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 42227 } } peers { permanent_uuid: "a1c8691b056b47b1b220ece9348bfaaf" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42587 } }
I20260812 02:41:34.291554 3798 leader_election.cc:288] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 [CANDIDATE]: Term 1 election: Requested vote from peers cc853a7e128a4d21b54e48ae80f8fb2c (127.3.127.129:42063), a1c8691b056b47b1b220ece9348bfaaf (127.3.127.131:42587)
I20260812 02:41:34.291960 3195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e100c58d8574272b0ff4a08683ca4f4" candidate_uuid: "c6f76c00992b44d193eb8ca26049a579" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc853a7e128a4d21b54e48ae80f8fb2c"
I20260812 02:41:34.291970 3682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e100c58d8574272b0ff4a08683ca4f4" candidate_uuid: "c6f76c00992b44d193eb8ca26049a579" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1c8691b056b47b1b220ece9348bfaaf"
I20260812 02:41:34.292088 3682 raft_consensus.cc:3037] T 2e100c58d8574272b0ff4a08683ca4f4 P a1c8691b056b47b1b220ece9348bfaaf [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:34.292110 3195 raft_consensus.cc:3037] T 2e100c58d8574272b0ff4a08683ca4f4 P cc853a7e128a4d21b54e48ae80f8fb2c [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:34.293467 3682 raft_consensus.cc:2454] T 2e100c58d8574272b0ff4a08683ca4f4 P a1c8691b056b47b1b220ece9348bfaaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6f76c00992b44d193eb8ca26049a579 in term 1.
I20260812 02:41:34.293525 3195 raft_consensus.cc:2454] T 2e100c58d8574272b0ff4a08683ca4f4 P cc853a7e128a4d21b54e48ae80f8fb2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6f76c00992b44d193eb8ca26049a579 in term 1.
I20260812 02:41:34.293882 3313 leader_election.cc:302] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a1c8691b056b47b1b220ece9348bfaaf, c6f76c00992b44d193eb8ca26049a579, cc853a7e128a4d21b54e48ae80f8fb2c; no voters:
I20260812 02:41:34.294134 3798 raft_consensus.cc:2781] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:34.294373 3798 raft_consensus.cc:686] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 [term 1 LEADER]: Becoming Leader. State: Replica: c6f76c00992b44d193eb8ca26049a579, State: Running, Role: LEADER
I20260812 02:41:34.294764 3798 consensus_queue.cc:237] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc853a7e128a4d21b54e48ae80f8fb2c" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 42063 } } peers { permanent_uuid: "c6f76c00992b44d193eb8ca26049a579" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 42227 } } peers { permanent_uuid: "a1c8691b056b47b1b220ece9348bfaaf" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42587 } }
I20260812 02:41:34.298183 2961 catalog_manager.cc:5495] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6f76c00992b44d193eb8ca26049a579 (127.3.127.130). New cstate: current_term: 1 leader_uuid: "c6f76c00992b44d193eb8ca26049a579" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc853a7e128a4d21b54e48ae80f8fb2c" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 42063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6f76c00992b44d193eb8ca26049a579" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 42227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1c8691b056b47b1b220ece9348bfaaf" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20260812 02:41:34.327243 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:34.323289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60750 (local address 127.3.127.130:42227)
0812 02:41:34.323431 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:34.323435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:34.323450 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:41:34.323567 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:34.323570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:34.323630 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:34.323727 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:41:34.323967 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.324494 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.325535 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.325781 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.325879 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:34.326282 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:41:34.326797 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:41:34.326907 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:34.327008 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 02:41:34.327076 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 02:41:34.333962 3682 raft_consensus.cc:1261] T 2e100c58d8574272b0ff4a08683ca4f4 P a1c8691b056b47b1b220ece9348bfaaf [term 1 FOLLOWER]: Refusing update from remote peer c6f76c00992b44d193eb8ca26049a579: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:41:34.333961 3195 raft_consensus.cc:1261] T 2e100c58d8574272b0ff4a08683ca4f4 P cc853a7e128a4d21b54e48ae80f8fb2c [term 1 FOLLOWER]: Refusing update from remote peer c6f76c00992b44d193eb8ca26049a579: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:41:34.334786 3800 consensus_queue.cc:1035] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc853a7e128a4d21b54e48ae80f8fb2c" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 42063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:41:34.335002 3798 consensus_queue.cc:1035] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1c8691b056b47b1b220ece9348bfaaf" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:41:34.342800 3807 mvcc.cc:205] Tried to move back new op lower bound from 7317514216783740928 to 7317514216634380288. Current Snapshot: MvccSnapshot[applied={T|T < 7317514216783740928}]
I20260812 02:41:34.344137 3808 mvcc.cc:205] Tried to move back new op lower bound from 7317514216783740928 to 7317514216634380288. Current Snapshot: MvccSnapshot[applied={T|T < 7317514216783740928}]
I20260812 02:41:34.344770 3809 mvcc.cc:205] Tried to move back new op lower bound from 7317514216783740928 to 7317514216634380288. Current Snapshot: MvccSnapshot[applied={T|T < 7317514216783740928}]
I20260812 02:41:34.354274 2960 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:41:34.354452 2960 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:41:34.357448 2960 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c80a395d4c1d4929965649caa88e978a: Sending DeleteTablet for 3 replicas of tablet 2e100c58d8574272b0ff4a08683ca4f4
I20260812 02:41:34.358464 3391 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e100c58d8574272b0ff4a08683ca4f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:41:34 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:60744
I20260812 02:41:34.358491 3649 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e100c58d8574272b0ff4a08683ca4f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:41:34 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:34286
I20260812 02:41:34.358541 3170 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e100c58d8574272b0ff4a08683ca4f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:41:34 UTC) from {username='kudu', principal='kudu/127.3.127.190@KRBTEST.COM'} at 127.0.0.1:47518
I20260812 02:41:34.359105 3822 tablet_replica.cc:330] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579: stopping tablet replica
I20260812 02:41:34.359266 3821 tablet_replica.cc:330] T 2e100c58d8574272b0ff4a08683ca4f4 P cc853a7e128a4d21b54e48ae80f8fb2c: stopping tablet replica
I20260812 02:41:34.359617 3821 raft_consensus.cc:2229] T 2e100c58d8574272b0ff4a08683ca4f4 P cc853a7e128a4d21b54e48ae80f8fb2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:41:34.359618 3822 raft_consensus.cc:2229] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:41:34.359865 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 3058
I20260812 02:41:34.362754 3823 tablet_replica.cc:330] T 2e100c58d8574272b0ff4a08683ca4f4 P a1c8691b056b47b1b220ece9348bfaaf: stopping tablet replica
I20260812 02:41:34.363224 3823 raft_consensus.cc:2229] T 2e100c58d8574272b0ff4a08683ca4f4 P a1c8691b056b47b1b220ece9348bfaaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:41:34.363698 3823 raft_consensus.cc:2258] T 2e100c58d8574272b0ff4a08683ca4f4 P a1c8691b056b47b1b220ece9348bfaaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:41:34.373631 3822 raft_consensus.cc:2258] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:41:34.377089 3822 ts_tablet_manager.cc:1905] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:41:34.377450 3823 ts_tablet_manager.cc:1905] T 2e100c58d8574272b0ff4a08683ca4f4 P a1c8691b056b47b1b220ece9348bfaaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:41:34.380213 3823 ts_tablet_manager.cc:1918] T 2e100c58d8574272b0ff4a08683ca4f4 P a1c8691b056b47b1b220ece9348bfaaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:41:34.380213 3822 ts_tablet_manager.cc:1918] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:41:34.380342 3822 log.cc:1196] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/2e100c58d8574272b0ff4a08683ca4f4
I20260812 02:41:34.380452 3823 log.cc:1196] T 2e100c58d8574272b0ff4a08683ca4f4 P a1c8691b056b47b1b220ece9348bfaaf: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/2e100c58d8574272b0ff4a08683ca4f4
I20260812 02:41:34.380854 3823 ts_tablet_manager.cc:1939] T 2e100c58d8574272b0ff4a08683ca4f4 P a1c8691b056b47b1b220ece9348bfaaf: Deleting consensus metadata
I20260812 02:41:34.381031 3822 ts_tablet_manager.cc:1939] T 2e100c58d8574272b0ff4a08683ca4f4 P c6f76c00992b44d193eb8ca26049a579: Deleting consensus metadata
I20260812 02:41:34.382500 2942 catalog_manager.cc:4841] TS a1c8691b056b47b1b220ece9348bfaaf (127.3.127.131:42587): tablet 2e100c58d8574272b0ff4a08683ca4f4 (table test-table [id=ee5c593187914de19ff703375464fa86]) successfully deleted
I20260812 02:41:34.382733 2941 catalog_manager.cc:4841] TS c6f76c00992b44d193eb8ca26049a579 (127.3.127.130:42227): tablet 2e100c58d8574272b0ff4a08683ca4f4 (table test-table [id=ee5c593187914de19ff703375464fa86]) successfully deleted
W20260812 02:41:34.390092 2941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.127.129:42063 (error 108)
I20260812 02:41:34.391215 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:34.390910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:42063 (local address 127.0.0.1:47526)
0812 02:41:34.391047 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:34.391120 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.129:42063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260812 02:41:34.391346 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 3284
W20260812 02:41:34.395332 2941 catalog_manager.cc:4584] TS cc853a7e128a4d21b54e48ae80f8fb2c (127.3.127.129:42063): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e100c58d8574272b0ff4a08683ca4f4: Network error: Client connection negotiation failed: client connection to 127.3.127.129:42063: connect: Connection refused (error 111)
I20260812 02:41:34.405807 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 3530
I20260812 02:41:34.423113 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 2917
2026-08-12T02:41:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2560 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T02:41:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:41:34Z Disabled control of system clock
I20260812 02:41:34.465099 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:39737
--webserver_interface=127.3.127.190
--webserver_port=0
--builtin_ntp_servers=127.3.127.148:43177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:39737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 02:41:34.587492 3834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:34.587826 3834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:34.587863 3834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:34.592095 3834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:34.592167 3834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:34.592182 3834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:34.592198 3834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:34.592213 3834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:34.597285 3834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:43177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:39737
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:39737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:34.598484 3834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:34.599922 3834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:34.609031 3846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:34.609052 3845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:34.609212 3834 server_base.cc:1034] running on GCE node
W20260812 02:41:34.609081 3849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:34.610074 3834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:34.611277 3834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:34.612452 3834 hybrid_clock.cc:648] HybridClock initialized: now 1786502494612418 us; error 48 us; skew 500 ppm
I20260812 02:41:34.614490 3834 webserver.cc:460] Webserver started at http://127.3.127.190:41975/ using document root <none> and password file <none>
I20260812 02:41:34.615084 3834 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:34.615154 3834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:34.615357 3834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:34.617617 3834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "501c9f0ad6ea48a0ac0a89c0358f3162"
format_stamp: "Formatted at 2026-08-12 02:41:34 on dist-test-slave-rmhg"
server_key: "de3d4ddde3b59cafaf370040bf1131df"
server_key_iv: "3819063230a475ccf43030233612fb24"
server_key_version: "encryptionkey@0"
I20260812 02:41:34.618208 3834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "501c9f0ad6ea48a0ac0a89c0358f3162"
format_stamp: "Formatted at 2026-08-12 02:41:34 on dist-test-slave-rmhg"
server_key: "de3d4ddde3b59cafaf370040bf1131df"
server_key_iv: "3819063230a475ccf43030233612fb24"
server_key_version: "encryptionkey@0"
I20260812 02:41:34.622102 3834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:41:34.624675 3856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:34.625976 3834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 02:41:34.626102 3834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "501c9f0ad6ea48a0ac0a89c0358f3162"
format_stamp: "Formatted at 2026-08-12 02:41:34 on dist-test-slave-rmhg"
server_key: "de3d4ddde3b59cafaf370040bf1131df"
server_key_iv: "3819063230a475ccf43030233612fb24"
server_key_version: "encryptionkey@0"
I20260812 02:41:34.626232 3834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:34.651759 3834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:34.652565 3834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:34.652781 3834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:34.660351 3834 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:39737
I20260812 02:41:34.660430 3942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:39737 every 8 connection(s)
I20260812 02:41:34.661455 3834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:34.663069 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 3834
I20260812 02:41:34.663213 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:41:34.663563 3582 external_mini_cluster.cc:1428] Setting key f41767f7c99fb685851d2a6a953b1bf5
I20260812 02:41:34.665596 3948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:34.671764 3948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162: Bootstrap starting.
I20260812 02:41:34.673558 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:34.665232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33668 (local address 127.3.127.190:39737)
0812 02:41:34.665980 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:34.666006 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:34.666047 (+ 41us) server_negotiation.cc:407] Connection header received
0812 02:41:34.666704 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:34.666721 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:34.667168 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:34.667490 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:34.667954 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.668986 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.669958 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.670474 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.670630 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:34.671594 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:34.671600 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:34.671618 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:34.672072 (+ 454us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:34.672138 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:34.672162 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:34.672253 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:34.672415 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 02:41:34.672706 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":177,"threads_started":1}
I20260812 02:41:34.674324 3948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:34.675354 3948 log.cc:824] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:34.677750 3948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162: No bootstrap required, opened a new log
I20260812 02:41:34.681819 3948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501c9f0ad6ea48a0ac0a89c0358f3162" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 39737 } }
I20260812 02:41:34.682102 3948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:34.682148 3948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501c9f0ad6ea48a0ac0a89c0358f3162, State: Initialized, Role: FOLLOWER
I20260812 02:41:34.682557 3948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501c9f0ad6ea48a0ac0a89c0358f3162" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 39737 } }
I20260812 02:41:34.682679 3948 raft_consensus.cc:388] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:34.682729 3948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:34.682806 3948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:34.683821 3948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501c9f0ad6ea48a0ac0a89c0358f3162" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 39737 } }
I20260812 02:41:34.684177 3948 leader_election.cc:302] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 501c9f0ad6ea48a0ac0a89c0358f3162; no voters:
I20260812 02:41:34.684520 3948 leader_election.cc:288] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:41:34.684695 3956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:34.684932 3956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162 [term 1 LEADER]: Becoming Leader. State: Replica: 501c9f0ad6ea48a0ac0a89c0358f3162, State: Running, Role: LEADER
I20260812 02:41:34.685287 3956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501c9f0ad6ea48a0ac0a89c0358f3162" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 39737 } }
I20260812 02:41:34.686005 3948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:34.687135 3959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "501c9f0ad6ea48a0ac0a89c0358f3162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501c9f0ad6ea48a0ac0a89c0358f3162" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 39737 } } }
I20260812 02:41:34.687280 3959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:34.687562 3960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 501c9f0ad6ea48a0ac0a89c0358f3162. Latest consensus state: current_term: 1 leader_uuid: "501c9f0ad6ea48a0ac0a89c0358f3162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501c9f0ad6ea48a0ac0a89c0358f3162" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 39737 } } }
I20260812 02:41:34.687677 3960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:34.687786 3971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:34.691061 3971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:34.696885 3971 catalog_manager.cc:1347] Generated new cluster ID: 19de6bd5548546038f54ca09fa1fe468
I20260812 02:41:34.696976 3971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:34.705040 3971 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:41:34.706027 3971 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:34.721565 3971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162: Generated new TSK 0
I20260812 02:41:34.722374 3971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:41:34.725466 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.129:0
--local_ip_for_outbound_sockets=127.3.127.129
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:39737
--builtin_ntp_servers=127.3.127.148:43177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 02:41:34.850613 3983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:34.850914 3983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:34.850955 3983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:34.855203 3983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:34.855275 3983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:34.855336 3983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.129
I20260812 02:41:34.860507 3983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:43177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.127.129
--webserver_port=0
--tserver_master_addrs=127.3.127.190:39737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.129
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:34.861819 3983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:34.863348 3983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:34.874318 3992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:34.874320 3997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:34.874320 3993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:34.874552 3983 server_base.cc:1034] running on GCE node
I20260812 02:41:34.875013 3983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:34.875746 3983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:34.876960 3983 hybrid_clock.cc:648] HybridClock initialized: now 1786502494876924 us; error 56 us; skew 500 ppm
I20260812 02:41:34.879273 3983 webserver.cc:460] Webserver started at http://127.3.127.129:41617/ using document root <none> and password file <none>
I20260812 02:41:34.879915 3983 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:34.879994 3983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:34.880226 3983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:34.882459 3983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-0/data/instance:
uuid: "4989c5b409464c53bd15d5dcc310e6dd"
format_stamp: "Formatted at 2026-08-12 02:41:34 on dist-test-slave-rmhg"
server_key: "7a5284233b3bcd8f37f0d8d7e65f1747"
server_key_iv: "f71a21c9b67d1d28c9c250defbfb62ee"
server_key_version: "encryptionkey@0"
I20260812 02:41:34.883033 3983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance:
uuid: "4989c5b409464c53bd15d5dcc310e6dd"
format_stamp: "Formatted at 2026-08-12 02:41:34 on dist-test-slave-rmhg"
server_key: "7a5284233b3bcd8f37f0d8d7e65f1747"
server_key_iv: "f71a21c9b67d1d28c9c250defbfb62ee"
server_key_version: "encryptionkey@0"
I20260812 02:41:34.887226 3983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:41:34.889990 4004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:34.891269 3983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 02:41:34.891388 3983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-0/wal
uuid: "4989c5b409464c53bd15d5dcc310e6dd"
format_stamp: "Formatted at 2026-08-12 02:41:34 on dist-test-slave-rmhg"
server_key: "7a5284233b3bcd8f37f0d8d7e65f1747"
server_key_iv: "f71a21c9b67d1d28c9c250defbfb62ee"
server_key_version: "encryptionkey@0"
I20260812 02:41:34.891495 3983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:34.911053 3983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:34.911873 3983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:34.912068 3983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:34.912719 3983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:41:34.913808 3983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:41:34.913858 3983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:34.913914 3983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:41:34.913940 3983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:34.923136 3983 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.129:44249
I20260812 02:41:34.923237 4181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.129:44249 every 8 connection(s)
I20260812 02:41:34.924134 3983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-0/data/info.pb
I20260812 02:41:34.933872 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:34.926153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.129:57263 (local address 127.3.127.190:39737)
0812 02:41:34.926293 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:34.926296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:34.927398 (+ 1102us) server_negotiation.cc:407] Connection header received
0812 02:41:34.928832 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:34.928835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:34.928892 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:34.928973 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:34.930005 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.930551 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.931815 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.932040 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.932136 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:34.933355 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:34.933358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:34.933360 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:34.933374 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:34.933395 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:34.933403 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:34.933463 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:34.933618 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 02:41:34.933711 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 02:41:34.934190 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 3983
I20260812 02:41:34.934322 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-0/wal/instance
I20260812 02:41:34.934653 3582 external_mini_cluster.cc:1428] Setting key 5078ae091111e7a51ddaf2fdcc753d6d
I20260812 02:41:34.934863 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:34.926554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:39737 (local address 127.3.127.129:57263)
0812 02:41:34.927259 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:34.927303 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:34.928587 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:34.929140 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:34.929177 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:34.929281 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:41:34.929845 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:34.929860 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.930693 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.930696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:34.931694 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:34.931705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:34.932198 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:34.932202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:34.932400 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:34.933055 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:41:34.933057 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:41:34.933101 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:41:34.933198 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:41:34.933473 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:41:34.933483 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:41:34.933489 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 02:41:34.933608 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 02:41:34.933942 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":138,"threads_started":1}
I20260812 02:41:34.936213 4183 heartbeater.cc:347] Connected to a master server at 127.3.127.190:39737
I20260812 02:41:34.936486 4183 heartbeater.cc:464] Registering TS with master...
I20260812 02:41:34.936831 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.130:0
--local_ip_for_outbound_sockets=127.3.127.130
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:39737
--builtin_ntp_servers=127.3.127.148:43177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 02:41:34.937189 4183 heartbeater.cc:511] Master 127.3.127.190:39737 requested a full tablet report, sending...
I20260812 02:41:34.938882 3881 ts_manager.cc:194] Registered new tserver with Master: 4989c5b409464c53bd15d5dcc310e6dd (127.3.127.129:44249)
I20260812 02:41:34.940413 3881 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.127.129:57263
W20260812 02:41:35.059633 4187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:35.059938 4187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:35.059978 4187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:35.064116 4187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:35.064205 4187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:35.064282 4187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.130
I20260812 02:41:35.069295 4187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:43177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.127.130
--webserver_port=0
--tserver_master_addrs=127.3.127.190:39737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.130
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:35.070467 4187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:35.071972 4187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:35.082881 4199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:35.082887 4196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:35.082887 4194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:35.083336 4187 server_base.cc:1034] running on GCE node
I20260812 02:41:35.083830 4187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:35.084625 4187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:35.085835 4187 hybrid_clock.cc:648] HybridClock initialized: now 1786502495085800 us; error 46 us; skew 500 ppm
I20260812 02:41:35.088222 4187 webserver.cc:460] Webserver started at http://127.3.127.130:34687/ using document root <none> and password file <none>
I20260812 02:41:35.088876 4187 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:35.088954 4187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:35.089227 4187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:35.091616 4187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-1/data/instance:
uuid: "b2e4523519f546c7afcbeb7b9b4322f9"
format_stamp: "Formatted at 2026-08-12 02:41:35 on dist-test-slave-rmhg"
server_key: "6aa41c50e9275bfb83a0db144f81222e"
server_key_iv: "a734c5574042cf96513f8783a5aa9ef7"
server_key_version: "encryptionkey@0"
I20260812 02:41:35.092268 4187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance:
uuid: "b2e4523519f546c7afcbeb7b9b4322f9"
format_stamp: "Formatted at 2026-08-12 02:41:35 on dist-test-slave-rmhg"
server_key: "6aa41c50e9275bfb83a0db144f81222e"
server_key_iv: "a734c5574042cf96513f8783a5aa9ef7"
server_key_version: "encryptionkey@0"
I20260812 02:41:35.096930 4187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 02:41:35.100096 4211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:35.101662 4187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:41:35.101804 4187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-1/wal
uuid: "b2e4523519f546c7afcbeb7b9b4322f9"
format_stamp: "Formatted at 2026-08-12 02:41:35 on dist-test-slave-rmhg"
server_key: "6aa41c50e9275bfb83a0db144f81222e"
server_key_iv: "a734c5574042cf96513f8783a5aa9ef7"
server_key_version: "encryptionkey@0"
I20260812 02:41:35.101945 4187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:35.125069 4187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:35.125993 4187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:35.126181 4187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:35.126785 4187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:41:35.127818 4187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:41:35.127872 4187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:35.127912 4187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:41:35.127928 4187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:35.137698 4187 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.130:34673
I20260812 02:41:35.137737 4402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.130:34673 every 8 connection(s)
I20260812 02:41:35.138725 4187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-1/data/info.pb
I20260812 02:41:35.145948 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 4187
I20260812 02:41:35.146083 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-1/wal/instance
I20260812 02:41:35.146409 3582 external_mini_cluster.cc:1428] Setting key 408e367ac30d71d1a98af13e65ab0804
I20260812 02:41:35.149003 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.127.131:0
--local_ip_for_outbound_sockets=127.3.127.131
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:39737
--builtin_ntp_servers=127.3.127.148:43177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 02:41:35.150111 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:35.140854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.130:37417 (local address 127.3.127.190:39737)
0812 02:41:35.140997 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:35.141000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:35.142403 (+ 1403us) server_negotiation.cc:407] Connection header received
0812 02:41:35.143957 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:35.143961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:35.144029 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:35.144114 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:35.145223 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.145802 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.147275 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.147585 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.147714 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:35.149524 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:35.149527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:35.149529 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:35.149546 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:35.149572 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:35.149580 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:35.149687 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:35.149869 (+ 182us) server_negotiation.cc:299] Negotiation successful
0812 02:41:35.149955 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 02:41:35.151237 4406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:35.141374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:39737 (local address 127.3.127.130:37417)
0812 02:41:35.142205 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:35.142256 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:35.143684 (+ 1428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:35.144345 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:35.144354 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:35.144460 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:41:35.145032 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:35.145047 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.145950 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.145953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:35.147102 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:35.147116 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.147798 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.147804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:35.148210 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:35.148975 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:41:35.148977 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:41:35.149032 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:41:35.149205 (+ 173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:41:35.149704 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:41:35.149717 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:41:35.149725 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 02:41:35.149861 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 02:41:35.150274 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":129,"threads_started":1}
I20260812 02:41:35.152683 4404 heartbeater.cc:347] Connected to a master server at 127.3.127.190:39737
I20260812 02:41:35.152956 4404 heartbeater.cc:464] Registering TS with master...
I20260812 02:41:35.153671 4404 heartbeater.cc:511] Master 127.3.127.190:39737 requested a full tablet report, sending...
I20260812 02:41:35.154990 3881 ts_manager.cc:194] Registered new tserver with Master: b2e4523519f546c7afcbeb7b9b4322f9 (127.3.127.130:34673)
I20260812 02:41:35.155846 3881 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.127.130:37417
W20260812 02:41:35.270540 4409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:35.270829 4409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:35.270869 4409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:35.275117 4409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:35.275184 4409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:35.275242 4409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.127.131
I20260812 02:41:35.280109 4409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:43177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.127.131
--webserver_port=0
--tserver_master_addrs=127.3.127.190:39737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.127.131
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:35.281232 4409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:35.282635 4409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:35.292856 4418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:35.292866 4420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:35.292866 4417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:35.293069 4409 server_base.cc:1034] running on GCE node
I20260812 02:41:35.293715 4409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:35.294368 4409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:35.295532 4409 hybrid_clock.cc:648] HybridClock initialized: now 1786502495295515 us; error 31 us; skew 500 ppm
I20260812 02:41:35.297526 4409 webserver.cc:460] Webserver started at http://127.3.127.131:41091/ using document root <none> and password file <none>
I20260812 02:41:35.298131 4409 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:35.298197 4409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:35.298399 4409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:35.300633 4409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-2/data/instance:
uuid: "72f388f04873413490db7770d9aa1f45"
format_stamp: "Formatted at 2026-08-12 02:41:35 on dist-test-slave-rmhg"
server_key: "aa8f374c0bc86f11bf984233b28f259b"
server_key_iv: "d2db0895abc9a0d1c6220ce26b01b464"
server_key_version: "encryptionkey@0"
I20260812 02:41:35.301259 4409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance:
uuid: "72f388f04873413490db7770d9aa1f45"
format_stamp: "Formatted at 2026-08-12 02:41:35 on dist-test-slave-rmhg"
server_key: "aa8f374c0bc86f11bf984233b28f259b"
server_key_iv: "d2db0895abc9a0d1c6220ce26b01b464"
server_key_version: "encryptionkey@0"
I20260812 02:41:35.305289 4409 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:41:35.308154 4432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:35.309538 4409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:41:35.309700 4409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-2/wal
uuid: "72f388f04873413490db7770d9aa1f45"
format_stamp: "Formatted at 2026-08-12 02:41:35 on dist-test-slave-rmhg"
server_key: "aa8f374c0bc86f11bf984233b28f259b"
server_key_iv: "d2db0895abc9a0d1c6220ce26b01b464"
server_key_version: "encryptionkey@0"
I20260812 02:41:35.309820 4409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:35.326983 4409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:35.327731 4409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:35.327913 4409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:35.328552 4409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:41:35.329588 4409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:41:35.329648 4409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:35.329698 4409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:41:35.329730 4409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:35.338693 4409 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.131:41771
I20260812 02:41:35.338769 4615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.131:41771 every 8 connection(s)
I20260812 02:41:35.339686 4409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-2/data/info.pb
I20260812 02:41:35.347886 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 4409
I20260812 02:41:35.348021 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-2/wal/instance
I20260812 02:41:35.348361 3582 external_mini_cluster.cc:1428] Setting key 80a51d6621e2453b95b2681998a50fb1
I20260812 02:41:35.350152 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:35.341700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:42541 (local address 127.3.127.190:39737)
0812 02:41:35.341861 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:35.341865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:35.342961 (+ 1096us) server_negotiation.cc:407] Connection header received
0812 02:41:35.344490 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:35.344494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:35.344554 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:35.344636 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:35.345812 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.346385 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.347801 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.348069 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.348182 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:35.349609 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:35.349613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:35.349615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:35.349631 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:35.349658 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:35.349668 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:35.349737 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:35.349929 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 02:41:35.350004 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 02:41:35.351276 4620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:35.342087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:39737 (local address 127.3.127.131:42541)
0812 02:41:35.342794 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:35.342838 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:35.344220 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:35.344840 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:35.344848 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:35.344950 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:41:35.345569 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:35.345585 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.346572 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.346575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:35.347647 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:35.347662 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.348257 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.348262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:35.348508 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:35.349277 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:41:35.349279 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:41:35.349342 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:41:35.349432 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:41:35.349745 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:41:35.349760 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:41:35.349765 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 02:41:35.349915 (+ 150us) client_negotiation.cc:240] Negotiation successful
0812 02:41:35.350248 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":119,"threads_started":1}
I20260812 02:41:35.352494 4616 heartbeater.cc:347] Connected to a master server at 127.3.127.190:39737
I20260812 02:41:35.352766 4616 heartbeater.cc:464] Registering TS with master...
I20260812 02:41:35.353394 4616 heartbeater.cc:511] Master 127.3.127.190:39737 requested a full tablet report, sending...
I20260812 02:41:35.354555 3881 ts_manager.cc:194] Registered new tserver with Master: 72f388f04873413490db7770d9aa1f45 (127.3.127.131:41771)
I20260812 02:41:35.355154 3881 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.127.131:42541
I20260812 02:41:35.362958 3582 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:41:35.368846 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:35.364983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33670 (local address 127.3.127.190:39737)
0812 02:41:35.365124 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:35.365129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:35.365332 (+ 203us) server_negotiation.cc:407] Connection header received
0812 02:41:35.365495 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:35.365498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:35.365546 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:35.365616 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:35.365911 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.366450 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.367338 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.367561 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.367651 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:35.368442 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:35.368446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:35.368448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:35.368461 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:35.368481 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:35.368489 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:35.368548 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:35.368643 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 02:41:35.368700 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 02:41:35.372047 3881 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:41:35.375315 3881 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:41:35.390450 4635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:35.384085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.131:41771 (local address 127.0.0.1:41768)
0812 02:41:35.384906 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:35.384965 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:35.385220 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:35.386022 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:35.386028 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:35.386054 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:35.386209 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:35.386216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.387455 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.387459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:35.388598 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:35.388609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.389851 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.389855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:35.389967 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:35.390010 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 02:41:35.390146 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 02:41:35.390289 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":115,"threads_started":1}
I20260812 02:41:35.390501 4636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:35.384253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41768 (local address 127.3.127.131:41771)
0812 02:41:35.384603 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:35.384616 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:35.385066 (+ 450us) server_negotiation.cc:407] Connection header received
0812 02:41:35.385416 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:35.385422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:35.385663 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:35.385800 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:35.386345 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.387357 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.388761 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.389457 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.389565 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:35.389787 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:35.390205 (+ 418us) server_negotiation.cc:299] Negotiation successful
0812 02:41:35.390325 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":99,"threads_started":1}
I20260812 02:41:35.390920 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:35.384892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:34673 (local address 127.0.0.1:44480)
0812 02:41:35.385812 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:35.385822 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:35.385894 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:35.386528 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:35.386531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:35.386549 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:35.386666 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:35.386673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.388108 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.388111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:35.389444 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:35.389456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.390512 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.390516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:35.390609 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:35.390634 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 02:41:35.390714 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 02:41:35.390784 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
I20260812 02:41:35.391110 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:35.384989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44046 (local address 127.3.127.129:44249)
0812 02:41:35.385491 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:35.385545 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:35.385880 (+ 335us) server_negotiation.cc:407] Connection header received
0812 02:41:35.386011 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:35.386016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:35.386218 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:35.386368 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:35.386605 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.387754 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.389272 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.390225 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.390322 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:35.390503 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:35.390840 (+ 337us) server_negotiation.cc:299] Negotiation successful
0812 02:41:35.390954 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":156,"threads_started":1}
I20260812 02:41:35.391402 4640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:35.385203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.3.127.130:34673)
0812 02:41:35.385651 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:35.385663 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:35.386010 (+ 347us) server_negotiation.cc:407] Connection header received
0812 02:41:35.386082 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:35.386086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:35.386237 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:35.386374 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:35.386792 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.387988 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.389582 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.390147 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.390250 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:35.390442 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:35.391140 (+ 698us) server_negotiation.cc:299] Negotiation successful
0812 02:41:35.391250 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":92,"threads_started":1}
I20260812 02:41:35.391556 4632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:35.384240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:44249 (local address 127.0.0.1:44046)
0812 02:41:35.385772 (+ 1532us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:35.385787 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:35.385893 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:35.386353 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:35.386355 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:35.386373 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:35.386474 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:35.386480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.387886 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.387890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:35.389115 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:35.389127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.390571 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.390575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:35.390681 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:35.390705 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 02:41:35.391345 (+ 640us) client_negotiation.cc:240] Negotiation successful
0812 02:41:35.391428 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1459,"mutex_wait_us":59,"thread_start_us":103,"threads_started":1}
I20260812 02:41:35.394172 4292 tablet_service.cc:1467] Processing CreateTablet for tablet 125c70d588624e4aa316300459f71cf8 (DEFAULT_TABLE table=test-table [id=a7424ee737694c579f114ccc2e06de7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:41:35.394196 4518 tablet_service.cc:1467] Processing CreateTablet for tablet 125c70d588624e4aa316300459f71cf8 (DEFAULT_TABLE table=test-table [id=a7424ee737694c579f114ccc2e06de7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:41:35.394235 4081 tablet_service.cc:1467] Processing CreateTablet for tablet 125c70d588624e4aa316300459f71cf8 (DEFAULT_TABLE table=test-table [id=a7424ee737694c579f114ccc2e06de7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:41:35.395288 4292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 125c70d588624e4aa316300459f71cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:35.395303 4518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 125c70d588624e4aa316300459f71cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:35.395303 4081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 125c70d588624e4aa316300459f71cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:35.402904 4643 tablet_bootstrap.cc:492] T 125c70d588624e4aa316300459f71cf8 P b2e4523519f546c7afcbeb7b9b4322f9: Bootstrap starting.
I20260812 02:41:35.403633 4644 tablet_bootstrap.cc:492] T 125c70d588624e4aa316300459f71cf8 P 4989c5b409464c53bd15d5dcc310e6dd: Bootstrap starting.
I20260812 02:41:35.405140 4643 tablet_bootstrap.cc:654] T 125c70d588624e4aa316300459f71cf8 P b2e4523519f546c7afcbeb7b9b4322f9: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:35.405524 4645 tablet_bootstrap.cc:492] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45: Bootstrap starting.
I20260812 02:41:35.405576 4644 tablet_bootstrap.cc:654] T 125c70d588624e4aa316300459f71cf8 P 4989c5b409464c53bd15d5dcc310e6dd: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:35.406078 4643 log.cc:824] T 125c70d588624e4aa316300459f71cf8 P b2e4523519f546c7afcbeb7b9b4322f9: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:35.406520 4644 log.cc:824] T 125c70d588624e4aa316300459f71cf8 P 4989c5b409464c53bd15d5dcc310e6dd: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:35.407788 4645 tablet_bootstrap.cc:654] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:35.408252 4643 tablet_bootstrap.cc:492] T 125c70d588624e4aa316300459f71cf8 P b2e4523519f546c7afcbeb7b9b4322f9: No bootstrap required, opened a new log
I20260812 02:41:35.408439 4643 ts_tablet_manager.cc:1397] T 125c70d588624e4aa316300459f71cf8 P b2e4523519f546c7afcbeb7b9b4322f9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 02:41:35.408660 4644 tablet_bootstrap.cc:492] T 125c70d588624e4aa316300459f71cf8 P 4989c5b409464c53bd15d5dcc310e6dd: No bootstrap required, opened a new log
I20260812 02:41:35.408859 4644 ts_tablet_manager.cc:1397] T 125c70d588624e4aa316300459f71cf8 P 4989c5b409464c53bd15d5dcc310e6dd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 02:41:35.408864 4645 log.cc:824] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:35.410923 4645 tablet_bootstrap.cc:492] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45: No bootstrap required, opened a new log
I20260812 02:41:35.411123 4645 ts_tablet_manager.cc:1397] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20260812 02:41:35.412063 4643 raft_consensus.cc:348] T 125c70d588624e4aa316300459f71cf8 P b2e4523519f546c7afcbeb7b9b4322f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f388f04873413490db7770d9aa1f45" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41771 } } peers { permanent_uuid: "4989c5b409464c53bd15d5dcc310e6dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 44249 } } peers { permanent_uuid: "b2e4523519f546c7afcbeb7b9b4322f9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34673 } }
I20260812 02:41:35.412310 4643 raft_consensus.cc:374] T 125c70d588624e4aa316300459f71cf8 P b2e4523519f546c7afcbeb7b9b4322f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:35.412364 4643 raft_consensus.cc:729] T 125c70d588624e4aa316300459f71cf8 P b2e4523519f546c7afcbeb7b9b4322f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e4523519f546c7afcbeb7b9b4322f9, State: Initialized, Role: FOLLOWER
I20260812 02:41:35.412827 4643 consensus_queue.cc:260] T 125c70d588624e4aa316300459f71cf8 P b2e4523519f546c7afcbeb7b9b4322f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f388f04873413490db7770d9aa1f45" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41771 } } peers { permanent_uuid: "4989c5b409464c53bd15d5dcc310e6dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 44249 } } peers { permanent_uuid: "b2e4523519f546c7afcbeb7b9b4322f9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34673 } }
I20260812 02:41:35.413133 4644 raft_consensus.cc:348] T 125c70d588624e4aa316300459f71cf8 P 4989c5b409464c53bd15d5dcc310e6dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f388f04873413490db7770d9aa1f45" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41771 } } peers { permanent_uuid: "4989c5b409464c53bd15d5dcc310e6dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 44249 } } peers { permanent_uuid: "b2e4523519f546c7afcbeb7b9b4322f9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34673 } }
I20260812 02:41:35.413442 4644 raft_consensus.cc:374] T 125c70d588624e4aa316300459f71cf8 P 4989c5b409464c53bd15d5dcc310e6dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:35.413496 4644 raft_consensus.cc:729] T 125c70d588624e4aa316300459f71cf8 P 4989c5b409464c53bd15d5dcc310e6dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4989c5b409464c53bd15d5dcc310e6dd, State: Initialized, Role: FOLLOWER
I20260812 02:41:35.413539 4404 heartbeater.cc:503] Master 127.3.127.190:39737 was elected leader, sending a full tablet report...
I20260812 02:41:35.413982 4643 ts_tablet_manager.cc:1428] T 125c70d588624e4aa316300459f71cf8 P b2e4523519f546c7afcbeb7b9b4322f9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 02:41:35.414016 4644 consensus_queue.cc:260] T 125c70d588624e4aa316300459f71cf8 P 4989c5b409464c53bd15d5dcc310e6dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f388f04873413490db7770d9aa1f45" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41771 } } peers { permanent_uuid: "4989c5b409464c53bd15d5dcc310e6dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 44249 } } peers { permanent_uuid: "b2e4523519f546c7afcbeb7b9b4322f9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34673 } }
I20260812 02:41:35.415035 4644 ts_tablet_manager.cc:1428] T 125c70d588624e4aa316300459f71cf8 P 4989c5b409464c53bd15d5dcc310e6dd: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260812 02:41:35.415421 4183 heartbeater.cc:503] Master 127.3.127.190:39737 was elected leader, sending a full tablet report...
I20260812 02:41:35.416414 4645 raft_consensus.cc:348] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f388f04873413490db7770d9aa1f45" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41771 } } peers { permanent_uuid: "4989c5b409464c53bd15d5dcc310e6dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 44249 } } peers { permanent_uuid: "b2e4523519f546c7afcbeb7b9b4322f9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34673 } }
I20260812 02:41:35.416703 4645 raft_consensus.cc:374] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:35.416769 4645 raft_consensus.cc:729] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f388f04873413490db7770d9aa1f45, State: Initialized, Role: FOLLOWER
I20260812 02:41:35.417266 4645 consensus_queue.cc:260] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f388f04873413490db7770d9aa1f45" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41771 } } peers { permanent_uuid: "4989c5b409464c53bd15d5dcc310e6dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 44249 } } peers { permanent_uuid: "b2e4523519f546c7afcbeb7b9b4322f9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34673 } }
I20260812 02:41:35.418000 4616 heartbeater.cc:503] Master 127.3.127.190:39737 was elected leader, sending a full tablet report...
I20260812 02:41:35.418354 4645 ts_tablet_manager.cc:1428] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
W20260812 02:41:35.425403 4184 tablet.cc:2369] T 125c70d588624e4aa316300459f71cf8 P 4989c5b409464c53bd15d5dcc310e6dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:41:35.521849 4651 raft_consensus.cc:482] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:41:35.522091 4651 raft_consensus.cc:504] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f388f04873413490db7770d9aa1f45" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41771 } } peers { permanent_uuid: "4989c5b409464c53bd15d5dcc310e6dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 44249 } } peers { permanent_uuid: "b2e4523519f546c7afcbeb7b9b4322f9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34673 } }
I20260812 02:41:35.523324 4651 leader_election.cc:288] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4989c5b409464c53bd15d5dcc310e6dd (127.3.127.129:44249), b2e4523519f546c7afcbeb7b9b4322f9 (127.3.127.130:34673)
I20260812 02:41:35.528425 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:35.523854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:52715 (local address 127.3.127.129:44249)
0812 02:41:35.524057 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:35.524062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:35.524075 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:41:35.524127 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:35.524130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:35.524181 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:35.524267 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:35.524555 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.525382 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.527097 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.527719 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.527849 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:35.527906 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:35.528206 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 02:41:35.528291 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260812 02:41:35.528425 4640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:35.523870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.127.131:59149 (local address 127.3.127.130:34673)
0812 02:41:35.524012 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:35.524017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:35.524166 (+ 149us) server_negotiation.cc:407] Connection header received
0812 02:41:35.524285 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:35.524288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:35.524338 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:35.524410 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:41:35.524668 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.525418 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.527097 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.527731 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.527851 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:35.527906 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:35.528206 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 02:41:35.528291 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 02:41:35.528422 4620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:35.523611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:44249 (local address 127.3.127.131:52715)
0812 02:41:35.523781 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:35.523797 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:35.523905 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:35.524291 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:35.524294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:35.524308 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:35.524424 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:35.524431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.525496 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.525500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:35.526949 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:35.526963 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.527850 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.527854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:35.528029 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:35.528080 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 02:41:35.528157 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 02:41:35.528243 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":121}
I20260812 02:41:35.528429 4652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:35.523649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.130:34673 (local address 127.3.127.131:59149)
0812 02:41:35.524090 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:35.524100 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 02:41:35.524199 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:41:35.524430 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:41:35.524433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:41:35.524447 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:41:35.524576 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:35.524581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.525513 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.525516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:35.526949 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:41:35.526963 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.527850 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.527854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:41:35.528037 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:35.528076 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 02:41:35.528176 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 02:41:35.528243 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":153,"thread_start_us":151,"threads_started":1}
I20260812 02:41:35.529202 4332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125c70d588624e4aa316300459f71cf8" candidate_uuid: "72f388f04873413490db7770d9aa1f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e4523519f546c7afcbeb7b9b4322f9" is_pre_election: true
I20260812 02:41:35.529210 4113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125c70d588624e4aa316300459f71cf8" candidate_uuid: "72f388f04873413490db7770d9aa1f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4989c5b409464c53bd15d5dcc310e6dd" is_pre_election: true
I20260812 02:41:35.529577 4332 raft_consensus.cc:2454] T 125c70d588624e4aa316300459f71cf8 P b2e4523519f546c7afcbeb7b9b4322f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72f388f04873413490db7770d9aa1f45 in term 0.
I20260812 02:41:35.529577 4113 raft_consensus.cc:2454] T 125c70d588624e4aa316300459f71cf8 P 4989c5b409464c53bd15d5dcc310e6dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72f388f04873413490db7770d9aa1f45 in term 0.
I20260812 02:41:35.530120 4438 leader_election.cc:302] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4989c5b409464c53bd15d5dcc310e6dd, 72f388f04873413490db7770d9aa1f45; no voters:
I20260812 02:41:35.530386 4651 raft_consensus.cc:2781] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:41:35.530478 4651 raft_consensus.cc:482] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:41:35.530531 4651 raft_consensus.cc:3037] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:35.531678 4651 raft_consensus.cc:504] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f388f04873413490db7770d9aa1f45" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41771 } } peers { permanent_uuid: "4989c5b409464c53bd15d5dcc310e6dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 44249 } } peers { permanent_uuid: "b2e4523519f546c7afcbeb7b9b4322f9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34673 } }
I20260812 02:41:35.532061 4651 leader_election.cc:288] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 [CANDIDATE]: Term 1 election: Requested vote from peers 4989c5b409464c53bd15d5dcc310e6dd (127.3.127.129:44249), b2e4523519f546c7afcbeb7b9b4322f9 (127.3.127.130:34673)
I20260812 02:41:35.532547 4332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125c70d588624e4aa316300459f71cf8" candidate_uuid: "72f388f04873413490db7770d9aa1f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e4523519f546c7afcbeb7b9b4322f9"
I20260812 02:41:35.532631 4113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125c70d588624e4aa316300459f71cf8" candidate_uuid: "72f388f04873413490db7770d9aa1f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4989c5b409464c53bd15d5dcc310e6dd"
I20260812 02:41:35.532737 4332 raft_consensus.cc:3037] T 125c70d588624e4aa316300459f71cf8 P b2e4523519f546c7afcbeb7b9b4322f9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:35.532804 4113 raft_consensus.cc:3037] T 125c70d588624e4aa316300459f71cf8 P 4989c5b409464c53bd15d5dcc310e6dd [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:35.533931 4332 raft_consensus.cc:2454] T 125c70d588624e4aa316300459f71cf8 P b2e4523519f546c7afcbeb7b9b4322f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72f388f04873413490db7770d9aa1f45 in term 1.
I20260812 02:41:35.533937 4113 raft_consensus.cc:2454] T 125c70d588624e4aa316300459f71cf8 P 4989c5b409464c53bd15d5dcc310e6dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72f388f04873413490db7770d9aa1f45 in term 1.
I20260812 02:41:35.534353 4433 leader_election.cc:302] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72f388f04873413490db7770d9aa1f45, b2e4523519f546c7afcbeb7b9b4322f9; no voters:
I20260812 02:41:35.534641 4651 raft_consensus.cc:2781] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:35.534948 4651 raft_consensus.cc:686] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 [term 1 LEADER]: Becoming Leader. State: Replica: 72f388f04873413490db7770d9aa1f45, State: Running, Role: LEADER
I20260812 02:41:35.535360 4651 consensus_queue.cc:237] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f388f04873413490db7770d9aa1f45" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41771 } } peers { permanent_uuid: "4989c5b409464c53bd15d5dcc310e6dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 44249 } } peers { permanent_uuid: "b2e4523519f546c7afcbeb7b9b4322f9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34673 } }
I20260812 02:41:35.538836 3879 catalog_manager.cc:5495] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72f388f04873413490db7770d9aa1f45 (127.3.127.131). New cstate: current_term: 1 leader_uuid: "72f388f04873413490db7770d9aa1f45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f388f04873413490db7770d9aa1f45" member_type: VOTER last_known_addr { host: "127.3.127.131" port: 41771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4989c5b409464c53bd15d5dcc310e6dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 44249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e4523519f546c7afcbeb7b9b4322f9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34673 } health_report { overall_health: UNKNOWN } } }
I20260812 02:41:35.585651 4636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:35.581938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41776 (local address 127.3.127.131:41771)
0812 02:41:35.582061 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:35.582065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:35.582078 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:41:35.582162 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:35.582165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:35.582226 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:35.582327 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:41:35.582586 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.583109 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.584073 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:35.584307 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:35.584399 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:35.584737 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:41:35.585225 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:41:35.585323 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:35.585436 (+ 113us) server_negotiation.cc:299] Negotiation successful
0812 02:41:35.585493 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260812 02:41:35.590660 4618 tablet.cc:2369] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:41:35.592343 4332 raft_consensus.cc:1261] T 125c70d588624e4aa316300459f71cf8 P b2e4523519f546c7afcbeb7b9b4322f9 [term 1 FOLLOWER]: Refusing update from remote peer 72f388f04873413490db7770d9aa1f45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:41:35.592343 4113 raft_consensus.cc:1261] T 125c70d588624e4aa316300459f71cf8 P 4989c5b409464c53bd15d5dcc310e6dd [term 1 FOLLOWER]: Refusing update from remote peer 72f388f04873413490db7770d9aa1f45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:41:35.593217 4651 consensus_queue.cc:1035] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e4523519f546c7afcbeb7b9b4322f9" member_type: VOTER last_known_addr { host: "127.3.127.130" port: 34673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:41:35.593436 4653 consensus_queue.cc:1035] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4989c5b409464c53bd15d5dcc310e6dd" member_type: VOTER last_known_addr { host: "127.3.127.129" port: 44249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:41:35.601831 4656 mvcc.cc:205] Tried to move back new op lower bound from 7317514221937713152 to 7317514221715636224. Current Snapshot: MvccSnapshot[applied={T|T < 7317514221937713152}]
I20260812 02:41:35.609501 4666 mvcc.cc:205] Tried to move back new op lower bound from 7317514221937713152 to 7317514221715636224. Current Snapshot: MvccSnapshot[applied={T|T < 7317514221937713152 or (T in {7317514221937713152})}]
I20260812 02:41:35.611071 3879 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:41:35.611239 3879 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:41:35.613622 3879 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 501c9f0ad6ea48a0ac0a89c0358f3162: Sending DeleteTablet for 3 replicas of tablet 125c70d588624e4aa316300459f71cf8
I20260812 02:41:35.614444 4081 tablet_service.cc:1514] Processing DeleteTablet for tablet 125c70d588624e4aa316300459f71cf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:41:35 UTC) from {username='slave'} at 127.0.0.1:44046
I20260812 02:41:35.614523 4518 tablet_service.cc:1514] Processing DeleteTablet for tablet 125c70d588624e4aa316300459f71cf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:41:35 UTC) from {username='slave'} at 127.0.0.1:41768
I20260812 02:41:35.615024 4292 tablet_service.cc:1514] Processing DeleteTablet for tablet 125c70d588624e4aa316300459f71cf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:41:35 UTC) from {username='slave'} at 127.0.0.1:44480
I20260812 02:41:35.615156 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 3983
I20260812 02:41:35.615653 4685 tablet_replica.cc:330] T 125c70d588624e4aa316300459f71cf8 P b2e4523519f546c7afcbeb7b9b4322f9: stopping tablet replica
I20260812 02:41:35.616068 4685 raft_consensus.cc:2229] T 125c70d588624e4aa316300459f71cf8 P b2e4523519f546c7afcbeb7b9b4322f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:41:35.616495 4685 raft_consensus.cc:2258] T 125c70d588624e4aa316300459f71cf8 P b2e4523519f546c7afcbeb7b9b4322f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:41:35.618304 4683 tablet_replica.cc:330] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45: stopping tablet replica
I20260812 02:41:35.618716 4683 raft_consensus.cc:2229] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:41:35.619160 4683 raft_consensus.cc:2258] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:41:35.621593 4685 ts_tablet_manager.cc:1905] T 125c70d588624e4aa316300459f71cf8 P b2e4523519f546c7afcbeb7b9b4322f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:41:35.624246 4685 ts_tablet_manager.cc:1918] T 125c70d588624e4aa316300459f71cf8 P b2e4523519f546c7afcbeb7b9b4322f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:41:35.624437 4685 log.cc:1196] T 125c70d588624e4aa316300459f71cf8 P b2e4523519f546c7afcbeb7b9b4322f9: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-1/wal/wals/125c70d588624e4aa316300459f71cf8
I20260812 02:41:35.624786 4685 ts_tablet_manager.cc:1939] T 125c70d588624e4aa316300459f71cf8 P b2e4523519f546c7afcbeb7b9b4322f9: Deleting consensus metadata
I20260812 02:41:35.626369 3857 catalog_manager.cc:4841] TS b2e4523519f546c7afcbeb7b9b4322f9 (127.3.127.130:34673): tablet 125c70d588624e4aa316300459f71cf8 (table test-table [id=a7424ee737694c579f114ccc2e06de7f]) successfully deleted
I20260812 02:41:35.626693 4683 ts_tablet_manager.cc:1905] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:41:35.630144 4683 ts_tablet_manager.cc:1918] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:41:35.630270 4683 log.cc:1196] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45: Deleting WAL directory at /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786502224378336-3582-0/minicluster-data/ts-2/wal/wals/125c70d588624e4aa316300459f71cf8
I20260812 02:41:35.630533 4683 ts_tablet_manager.cc:1939] T 125c70d588624e4aa316300459f71cf8 P 72f388f04873413490db7770d9aa1f45: Deleting consensus metadata
I20260812 02:41:35.633889 3857 catalog_manager.cc:4841] TS 72f388f04873413490db7770d9aa1f45 (127.3.127.131:41771): tablet 125c70d588624e4aa316300459f71cf8 (table test-table [id=a7424ee737694c579f114ccc2e06de7f]) successfully deleted
W20260812 02:41:35.643200 3858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.127.129:44249 (error 108)
I20260812 02:41:35.643532 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 4187
I20260812 02:41:35.658324 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:35.657925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.129:44249 (local address 127.0.0.1:44058)
0812 02:41:35.658089 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 02:41:35.658200 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.129:44249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260812 02:41:35.658641 3858 catalog_manager.cc:4584] TS 4989c5b409464c53bd15d5dcc310e6dd (127.3.127.129:44249): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 125c70d588624e4aa316300459f71cf8: Network error: Client connection negotiation failed: client connection to 127.3.127.129:44249: connect: Connection refused (error 111)
I20260812 02:41:35.668161 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 4409
I20260812 02:41:35.687975 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 3834
2026-08-12T02:41:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1265 ms)
[----------] 6 tests from EncryptionPolicyTest (11961 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:41:35 dist-test-slave-rmhg krb5kdc[4692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:41:35 dist-test-slave-rmhg krb5kdc[4692](info): set up 2 sockets
Aug 12 02:41:35 dist-test-slave-rmhg krb5kdc[4692](info): commencing operation
krb5kdc: starting...
W20260812 02:41:36.870450 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:41:36 dist-test-slave-rmhg krb5kdc[4692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:41:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:41:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:41:37.000582 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:37.123325 4716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:37.123646 4716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:37.123692 4716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:37.127967 4716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:37.128052 4716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:37.128072 4716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:37.128087 4716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:37.128101 4716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:37.133849 4716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:37.135004 4716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:37.136029 4716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:37.144913 4726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:37.144944 4735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:37.144953 4727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:37.145376 4716 server_base.cc:1034] running on GCE node
I20260812 02:41:37.146034 4716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:37.147259 4716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:37.148437 4716 hybrid_clock.cc:648] HybridClock initialized: now 1786502497148411 us; error 41 us; skew 500 ppm
Aug 12 02:41:37 dist-test-slave-rmhg krb5kdc[4692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:37.157464 4716 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:41:37.158609 4716 webserver.cc:460] Webserver started at http://127.0.0.1:41867/ using document root <none> and password file <none>
I20260812 02:41:37.159188 4716 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:37.159258 4716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:37.159452 4716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:37.161937 4716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "62920738da5b46378b43fbf1711e3426"
format_stamp: "Formatted at 2026-08-12 02:41:37 on dist-test-slave-rmhg"
server_key: "37c1294f67888d1da7ace504bdcc3c6d"
server_key_iv: "186fc9f47a912097024b867e5319e21f"
server_key_version: "encryptionkey@0"
I20260812 02:41:37.162513 4716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "62920738da5b46378b43fbf1711e3426"
format_stamp: "Formatted at 2026-08-12 02:41:37 on dist-test-slave-rmhg"
server_key: "37c1294f67888d1da7ace504bdcc3c6d"
server_key_iv: "186fc9f47a912097024b867e5319e21f"
server_key_version: "encryptionkey@0"
I20260812 02:41:37.166740 4716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:41:37.169750 4743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:37.171216 4716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:41:37.171355 4716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "62920738da5b46378b43fbf1711e3426"
format_stamp: "Formatted at 2026-08-12 02:41:37 on dist-test-slave-rmhg"
server_key: "37c1294f67888d1da7ace504bdcc3c6d"
server_key_iv: "186fc9f47a912097024b867e5319e21f"
server_key_version: "encryptionkey@0"
I20260812 02:41:37.171500 4716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:37.187090 4716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:37.190681 4716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:37.190886 4716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:37.198348 4716 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36973
I20260812 02:41:37.198370 4829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36973 every 8 connection(s)
I20260812 02:41:37.199445 4716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:37.202471 4832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:37.208261 4832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426: Bootstrap starting.
I20260812 02:41:37.208890 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 4716
I20260812 02:41:37.209018 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:41:37.209350 3582 external_mini_cluster.cc:1428] Setting key 1deb03654da2a7378d86cf2e97e61647
I20260812 02:41:37.211171 4832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:37.211910 4832 log.cc:824] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:37.214243 4832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426: No bootstrap required, opened a new log
Aug 12 02:41:37 dist-test-slave-rmhg krb5kdc[4692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 02:41:37.218875 4832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62920738da5b46378b43fbf1711e3426" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } }
I20260812 02:41:37.219166 4832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:37.219223 4832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62920738da5b46378b43fbf1711e3426, State: Initialized, Role: FOLLOWER
I20260812 02:41:37.219805 4832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62920738da5b46378b43fbf1711e3426" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } }
I20260812 02:41:37.219992 4832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:37.220069 4832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:37.220180 4832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:37.221357 4832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62920738da5b46378b43fbf1711e3426" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } }
I20260812 02:41:37.221779 4832 leader_election.cc:302] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62920738da5b46378b43fbf1711e3426; no voters:
I20260812 02:41:37.222211 4832 leader_election.cc:288] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:41:37.222311 4845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:37.222549 4845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426 [term 1 LEADER]: Becoming Leader. State: Replica: 62920738da5b46378b43fbf1711e3426, State: Running, Role: LEADER
I20260812 02:41:37.222918 4845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62920738da5b46378b43fbf1711e3426" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } }
I20260812 02:41:37.223786 4832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:37.224664 4850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62920738da5b46378b43fbf1711e3426. Latest consensus state: current_term: 1 leader_uuid: "62920738da5b46378b43fbf1711e3426" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62920738da5b46378b43fbf1711e3426" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } } }
I20260812 02:41:37.224808 4850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:37.225195 4848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62920738da5b46378b43fbf1711e3426" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62920738da5b46378b43fbf1711e3426" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } } }
I20260812 02:41:37.225297 4848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:37.226645 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:37.210639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40259 (local address 127.0.0.1:36973)
0812 02:41:37.211102 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:37.211118 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:37.211160 (+ 42us) server_negotiation.cc:407] Connection header received
0812 02:41:37.211852 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:37.211871 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:37.212265 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:37.212669 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:37.213586 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:37.214689 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:37.215883 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:37.216394 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:37.216522 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:37.219545 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:37.219552 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:37.219572 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:37.219613 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:37.221716 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:37.222412 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:37.222420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:37.222427 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:37.222596 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:37.223053 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:37.223056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:37.223057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:37.223619 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:37.224508 (+ 889us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:37.224716 (+ 208us) server_negotiation.cc:299] Negotiation successful
0812 02:41:37.224994 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":134,"threads_started":1}
W20260812 02:41:37.229483 4869 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:41:37.229589 4869 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:41:37.229665 4858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:37.229931 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:41:37.230587 4858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:37.237510 4858 catalog_manager.cc:1347] Generated new cluster ID: 528777cceaa64bda86ee60d024243854
I20260812 02:41:37.237591 4858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:37.241115 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:37.232284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45283 (local address 127.0.0.1:36973)
0812 02:41:37.232397 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:37.232401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:37.232689 (+ 288us) server_negotiation.cc:407] Connection header received
0812 02:41:37.232813 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:37.232817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:37.232871 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:37.232964 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:37.233950 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:37.234701 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:37.235795 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:37.236020 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:37.236132 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:37.237692 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:37.237696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:37.237698 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:37.237724 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:37.239321 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:37.239883 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:37.239886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:37.239888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:37.239963 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:37.240271 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:37.240274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:37.240276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:37.240461 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:37.240593 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:37.240854 (+ 261us) server_negotiation.cc:299] Negotiation successful
0812 02:41:37.240950 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 02:41:37.251967 4858 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:41:37.252936 4858 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:37.258965 4858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62920738da5b46378b43fbf1711e3426: Generated new TSK 0
I20260812 02:41:37.259737 4858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:41:37.268044 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 4716
W20260812 02:41:37.286620 4688 connection.cc:535] client connection to 127.0.0.1:36973 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36973): Connection reset by peer (error 104)
2026-08-12T02:41:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1581 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:41:37 dist-test-slave-rmhg krb5kdc[4885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:41:37 dist-test-slave-rmhg krb5kdc[4885](info): set up 2 sockets
Aug 12 02:41:37 dist-test-slave-rmhg krb5kdc[4885](info): commencing operation
krb5kdc: starting...
W20260812 02:41:38.445726 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:41:38 dist-test-slave-rmhg krb5kdc[4885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:41:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:41:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:41:38.582536 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33335
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:38.703433 4910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:38.703733 4910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:38.703794 4910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:38.707894 4910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:38.707984 4910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:38.708010 4910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:38.708032 4910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:38.708055 4910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:38.713647 4910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33335
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:38.714851 4910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:38.715919 4910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:41:38.724886 4910 server_base.cc:1034] running on GCE node
W20260812 02:41:38.724844 4918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:38.724836 4922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:38.724843 4920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:38.725778 4910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:38.726931 4910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:38.728101 4910 hybrid_clock.cc:648] HybridClock initialized: now 1786502498728084 us; error 32 us; skew 500 ppm
Aug 12 02:41:38 dist-test-slave-rmhg krb5kdc[4885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:38.731019 4910 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:41:38.732015 4910 webserver.cc:460] Webserver started at http://127.0.0.1:46703/ using document root <none> and password file <none>
I20260812 02:41:38.732568 4910 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:38.732631 4910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:38.732815 4910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:38.735018 4910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "9d272f08627a449090828455ba213606"
format_stamp: "Formatted at 2026-08-12 02:41:38 on dist-test-slave-rmhg"
server_key: "b031669d523ec0e10fe8032101b7fa1d"
server_key_iv: "d9cec56ec4e099ce106ae9ca6c7668a7"
server_key_version: "encryptionkey@0"
I20260812 02:41:38.735554 4910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "9d272f08627a449090828455ba213606"
format_stamp: "Formatted at 2026-08-12 02:41:38 on dist-test-slave-rmhg"
server_key: "b031669d523ec0e10fe8032101b7fa1d"
server_key_iv: "d9cec56ec4e099ce106ae9ca6c7668a7"
server_key_version: "encryptionkey@0"
I20260812 02:41:38.739377 4910 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260812 02:41:38.741926 4937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:38.743257 4910 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 02:41:38.743377 4910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "9d272f08627a449090828455ba213606"
format_stamp: "Formatted at 2026-08-12 02:41:38 on dist-test-slave-rmhg"
server_key: "b031669d523ec0e10fe8032101b7fa1d"
server_key_iv: "d9cec56ec4e099ce106ae9ca6c7668a7"
server_key_version: "encryptionkey@0"
I20260812 02:41:38.743476 4910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:38.755590 4910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:38.758816 4910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:38.758986 4910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:38.767035 4910 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33335
I20260812 02:41:38.767100 5022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33335 every 8 connection(s)
I20260812 02:41:38.768132 4910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:38.770644 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 4910
I20260812 02:41:38.770783 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:41:38.771142 3582 external_mini_cluster.cc:1428] Setting key 9a1b4cb77814eacb25c2290b2b9dd037
I20260812 02:41:38.771472 5023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:38.778364 5023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606: Bootstrap starting.
I20260812 02:41:38.780531 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:38.772651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35305 (local address 127.0.0.1:33335)
0812 02:41:38.773209 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:38.773226 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:38.773266 (+ 40us) server_negotiation.cc:407] Connection header received
0812 02:41:38.773951 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:38.773969 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:38.774343 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:38.774666 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:38.775103 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:38.776137 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:38.777080 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:38.777666 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:38.777825 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:38.778748 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:38.778753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:38.778767 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:38.779188 (+ 421us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:38.779249 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:38.779274 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:38.779394 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:38.779600 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 02:41:38.779750 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":179,"threads_started":1}
I20260812 02:41:38.781032 5023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:38.782202 5023 log.cc:824] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:38.783763 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:41:38.784420 5023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606: No bootstrap required, opened a new log
I20260812 02:41:38.789062 5023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d272f08627a449090828455ba213606" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } }
I20260812 02:41:38.789413 5023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:38.789482 5023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d272f08627a449090828455ba213606, State: Initialized, Role: FOLLOWER
I20260812 02:41:38.790023 5023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d272f08627a449090828455ba213606" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } }
I20260812 02:41:38.790222 5023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:38.790318 5023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:38.790354 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:38.785893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35813 (local address 127.0.0.1:33335)
0812 02:41:38.786015 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:38.786019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:38.786170 (+ 151us) server_negotiation.cc:407] Connection header received
0812 02:41:38.786312 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:38.786315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:38.786356 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:38.786411 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:38.786754 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:38.787280 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:38.788418 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:38.788653 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:38.788732 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:38.789823 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:38.789827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:38.789831 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:38.789846 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:38.789878 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:38.789891 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:38.789966 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:38.790106 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 02:41:38.790143 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 02:41:38.790436 5023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:38.791438 5023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d272f08627a449090828455ba213606" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } }
I20260812 02:41:38.791796 5023 leader_election.cc:302] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d272f08627a449090828455ba213606; no voters:
I20260812 02:41:38.792150 5023 leader_election.cc:288] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:41:38.792259 5037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:38.792474 5037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606 [term 1 LEADER]: Becoming Leader. State: Replica: 9d272f08627a449090828455ba213606, State: Running, Role: LEADER
I20260812 02:41:38.792853 5037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d272f08627a449090828455ba213606" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } }
I20260812 02:41:38.793496 5023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:38.794782 5041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d272f08627a449090828455ba213606. Latest consensus state: current_term: 1 leader_uuid: "9d272f08627a449090828455ba213606" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d272f08627a449090828455ba213606" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } } }
I20260812 02:41:38.794816 5039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d272f08627a449090828455ba213606" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d272f08627a449090828455ba213606" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } } }
I20260812 02:41:38.794947 5041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:38.794950 5039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:38.795320 5055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:38.798821 5055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:38.804966 5055 catalog_manager.cc:1347] Generated new cluster ID: 35ef4812cb7b4c6c97c6fa07a2cffe5a
I20260812 02:41:38.805074 5055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:38.828075 5055 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:41:38.829704 5055 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:38.836529 5055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d272f08627a449090828455ba213606: Generated new TSK 0
I20260812 02:41:38.837574 5055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:41:38.852123 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 4910
W20260812 02:41:38.865332 4883 connection.cc:535] client connection to 127.0.0.1:33335 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33335): Connection reset by peer (error 104)
2026-08-12T02:41:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1578 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:41:38 dist-test-slave-rmhg krb5kdc[5079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:41:38 dist-test-slave-rmhg krb5kdc[5079](info): set up 2 sockets
Aug 12 02:41:38 dist-test-slave-rmhg krb5kdc[5079](info): commencing operation
krb5kdc: starting...
W20260812 02:41:40.013818 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:41:40 dist-test-slave-rmhg krb5kdc[5079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:41:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:41:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:41:40.148886 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:40.269385 5098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:40.269701 5098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:40.269745 5098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:40.273870 5098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:40.273952 5098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:40.273969 5098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:40.273985 5098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:40.274003 5098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:40.279644 5098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42867
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:40.280882 5098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:40.281997 5098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:40.291122 5110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:40.291124 5105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:40.291244 5098 server_base.cc:1034] running on GCE node
W20260812 02:41:40.291145 5106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:40.291977 5098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:40.293184 5098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:40.294377 5098 hybrid_clock.cc:648] HybridClock initialized: now 1786502500294352 us; error 33 us; skew 500 ppm
Aug 12 02:41:40 dist-test-slave-rmhg krb5kdc[5079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:40.297221 5098 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:41:40.298174 5098 webserver.cc:460] Webserver started at http://127.0.0.1:36679/ using document root <none> and password file <none>
I20260812 02:41:40.298732 5098 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:40.298795 5098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:40.298976 5098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:40.301183 5098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "c6c9e9d91de344d3b8f99b4f732c25aa"
format_stamp: "Formatted at 2026-08-12 02:41:40 on dist-test-slave-rmhg"
server_key: "ce42ea0433ff9f24ed41accab505400e"
server_key_iv: "5f6c4ec79792745d1c70260b0252c33e"
server_key_version: "encryptionkey@0"
I20260812 02:41:40.301801 5098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "c6c9e9d91de344d3b8f99b4f732c25aa"
format_stamp: "Formatted at 2026-08-12 02:41:40 on dist-test-slave-rmhg"
server_key: "ce42ea0433ff9f24ed41accab505400e"
server_key_iv: "5f6c4ec79792745d1c70260b0252c33e"
server_key_version: "encryptionkey@0"
I20260812 02:41:40.306301 5098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:41:40.309060 5123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:40.310525 5098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:41:40.310705 5098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "c6c9e9d91de344d3b8f99b4f732c25aa"
format_stamp: "Formatted at 2026-08-12 02:41:40 on dist-test-slave-rmhg"
server_key: "ce42ea0433ff9f24ed41accab505400e"
server_key_iv: "5f6c4ec79792745d1c70260b0252c33e"
server_key_version: "encryptionkey@0"
I20260812 02:41:40.310835 5098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:40.315755 5098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:40.319005 5098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:40.319191 5098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:40.326707 5098 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42867
I20260812 02:41:40.326728 5205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42867 every 8 connection(s)
I20260812 02:41:40.327749 5098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:40.330852 5206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:40.336809 5206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa: Bootstrap starting.
I20260812 02:41:40.337451 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 5098
I20260812 02:41:40.337600 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:41:40.337929 3582 external_mini_cluster.cc:1428] Setting key e468c02e19d5b50ec76b86e09f2f6a24
I20260812 02:41:40.339887 5206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:40.340643 5206 log.cc:824] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:40.342962 5206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa: No bootstrap required, opened a new log
I20260812 02:41:40.343325 5214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:40.339353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50171 (local address 127.0.0.1:42867)
0812 02:41:40.339797 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:40.339813 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:40.339851 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:41:40.340543 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:40.340569 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:40.340895 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:40.341292 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:40.342040 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:40.342045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:40.342057 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:40.342069 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:40.342129 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:40.342150 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:40.342250 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:40.342380 (+ 130us) server_negotiation.cc:299] Negotiation successful
0812 02:41:40.342543 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":112,"threads_started":1}
I20260812 02:41:40.346289 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:41:40.347015 5206 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c9e9d91de344d3b8f99b4f732c25aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42867 } }
I20260812 02:41:40.347366 5206 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:40.347440 5206 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c9e9d91de344d3b8f99b4f732c25aa, State: Initialized, Role: FOLLOWER
I20260812 02:41:40.348018 5206 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c9e9d91de344d3b8f99b4f732c25aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42867 } }
I20260812 02:41:40.348212 5206 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:40.348287 5206 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:40.348395 5206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:40.349685 5206 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c9e9d91de344d3b8f99b4f732c25aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42867 } }
I20260812 02:41:40.350111 5206 leader_election.cc:302] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6c9e9d91de344d3b8f99b4f732c25aa; no voters:
I20260812 02:41:40.350492 5206 leader_election.cc:288] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:41:40.350612 5214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:40.348898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33063 (local address 127.0.0.1:42867)
0812 02:41:40.349031 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:40.349036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:40.349289 (+ 253us) server_negotiation.cc:407] Connection header received
0812 02:41:40.349396 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:40.349399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:40.349442 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:40.349536 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:40.350201 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:40.350204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:40.350207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:40.350220 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:40.350252 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:40.350265 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:40.350353 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:40.350416 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 02:41:40.350457 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 02:41:40.350728 5227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:40.350941 5227 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa [term 1 LEADER]: Becoming Leader. State: Replica: c6c9e9d91de344d3b8f99b4f732c25aa, State: Running, Role: LEADER
I20260812 02:41:40.351289 5227 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c9e9d91de344d3b8f99b4f732c25aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42867 } }
I20260812 02:41:40.351985 5206 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:40.352555 5229 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6c9e9d91de344d3b8f99b4f732c25aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c9e9d91de344d3b8f99b4f732c25aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42867 } } }
I20260812 02:41:40.352706 5229 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:40.352566 5227 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6c9e9d91de344d3b8f99b4f732c25aa. Latest consensus state: current_term: 1 leader_uuid: "c6c9e9d91de344d3b8f99b4f732c25aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c9e9d91de344d3b8f99b4f732c25aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42867 } } }
I20260812 02:41:40.353029 5227 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:40.353120 5235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:40.356550 5235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:40.363336 5235 catalog_manager.cc:1347] Generated new cluster ID: 7a1b32fa61bd4dbb80e0adbb8bfed2cf
I20260812 02:41:40.363432 5235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:40.397716 5235 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:41:40.398095 5235 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:40.402639 5235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6c9e9d91de344d3b8f99b4f732c25aa: Generated new TSK 0
I20260812 02:41:40.403405 5235 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:41:40.410744 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 5098
2026-08-12T02:41:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1561 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:41:40 dist-test-slave-rmhg krb5kdc[5254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:41:40 dist-test-slave-rmhg krb5kdc[5254](info): set up 2 sockets
Aug 12 02:41:40 dist-test-slave-rmhg krb5kdc[5254](info): commencing operation
krb5kdc: starting...
W20260812 02:41:41.596669 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:41:41 dist-test-slave-rmhg krb5kdc[5254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:41:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:41:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:41:41.731877 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35129
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:41.850430 5273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:41.850729 5273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:41.850792 5273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:41.854920 5273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:41.854995 5273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:41.855020 5273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:41.855043 5273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:41.855073 5273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:41.860585 5273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:41.861735 5273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:41.862792 5273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:41.871587 5284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:41.871618 5279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:41.871706 5282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:41.871871 5273 server_base.cc:1034] running on GCE node
I20260812 02:41:41.872488 5273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:41.873718 5273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:41.874886 5273 hybrid_clock.cc:648] HybridClock initialized: now 1786502501874863 us; error 38 us; skew 500 ppm
Aug 12 02:41:41 dist-test-slave-rmhg krb5kdc[5254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:41.877710 5273 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:41:41.878638 5273 webserver.cc:460] Webserver started at http://127.0.0.1:37281/ using document root <none> and password file <none>
I20260812 02:41:41.879199 5273 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:41.879269 5273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:41.879487 5273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:41.881693 5273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "87b1f20b6b8d4c59a36813e2c813a8e1"
format_stamp: "Formatted at 2026-08-12 02:41:41 on dist-test-slave-rmhg"
server_key: "ca4c6a26de9c8890be8f76b22416f382"
server_key_iv: "25dee0e187f40ad2ff6a650c9705245b"
server_key_version: "encryptionkey@0"
I20260812 02:41:41.882248 5273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "87b1f20b6b8d4c59a36813e2c813a8e1"
format_stamp: "Formatted at 2026-08-12 02:41:41 on dist-test-slave-rmhg"
server_key: "ca4c6a26de9c8890be8f76b22416f382"
server_key_iv: "25dee0e187f40ad2ff6a650c9705245b"
server_key_version: "encryptionkey@0"
I20260812 02:41:41.886023 5273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:41:41.888420 5293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:41.889684 5273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 02:41:41.889801 5273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "87b1f20b6b8d4c59a36813e2c813a8e1"
format_stamp: "Formatted at 2026-08-12 02:41:41 on dist-test-slave-rmhg"
server_key: "ca4c6a26de9c8890be8f76b22416f382"
server_key_iv: "25dee0e187f40ad2ff6a650c9705245b"
server_key_version: "encryptionkey@0"
I20260812 02:41:41.889894 5273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:41.920079 5273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:41.926020 5273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:41.926234 5273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:41.934388 5375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35129 every 8 connection(s)
I20260812 02:41:41.934401 5273 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35129
I20260812 02:41:41.935791 5273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:41.939261 5376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:41.939862 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 5273
I20260812 02:41:41.939992 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:41:41.940305 3582 external_mini_cluster.cc:1428] Setting key e066400cf4b6a2ba94a55c980e3cd9a8
I20260812 02:41:41.945717 5376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1: Bootstrap starting.
I20260812 02:41:41.947979 5376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:41.948917 5376 log.cc:824] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:41.951387 5376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1: No bootstrap required, opened a new log
I20260812 02:41:41.955761 5376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1f20b6b8d4c59a36813e2c813a8e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35129 } }
I20260812 02:41:41.955992 5376 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:41.956027 5376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b1f20b6b8d4c59a36813e2c813a8e1, State: Initialized, Role: FOLLOWER
I20260812 02:41:41.956463 5376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1f20b6b8d4c59a36813e2c813a8e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35129 } }
I20260812 02:41:41.956595 5376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:41.956672 5376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:41.956772 5376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1 [term 0 FOLLOWER]: Advancing to term 1
Aug 12 02:41:41 dist-test-slave-rmhg krb5kdc[5254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 02:41:41.957909 5376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1f20b6b8d4c59a36813e2c813a8e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35129 } }
I20260812 02:41:41.958299 5376 leader_election.cc:302] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87b1f20b6b8d4c59a36813e2c813a8e1; no voters:
I20260812 02:41:41.958644 5376 leader_election.cc:288] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:41:41.958735 5382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:41.958904 5382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1 [term 1 LEADER]: Becoming Leader. State: Replica: 87b1f20b6b8d4c59a36813e2c813a8e1, State: Running, Role: LEADER
I20260812 02:41:41.959285 5382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1f20b6b8d4c59a36813e2c813a8e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35129 } }
I20260812 02:41:41.960040 5376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:41.960583 5382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87b1f20b6b8d4c59a36813e2c813a8e1. Latest consensus state: current_term: 1 leader_uuid: "87b1f20b6b8d4c59a36813e2c813a8e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1f20b6b8d4c59a36813e2c813a8e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35129 } } }
I20260812 02:41:41.960975 5382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:41.961630 5384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87b1f20b6b8d4c59a36813e2c813a8e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1f20b6b8d4c59a36813e2c813a8e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35129 } } }
I20260812 02:41:41.961793 5384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:41.963683 5380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:41.941760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51775 (local address 127.0.0.1:35129)
0812 02:41:41.942238 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:41.942253 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:41.948552 (+ 6299us) server_negotiation.cc:407] Connection header received
0812 02:41:41.949595 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:41.949623 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:41.950063 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:41.950444 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:41.951414 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:41.952931 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:41.953906 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:41.954656 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:41.954838 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:41.957643 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:41.957648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:41.957661 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:41.957691 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:41.959635 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:41.960096 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:41.960102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:41.960109 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:41.960266 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:41.960598 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:41.960602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:41.960603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:41.961191 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:41.961393 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:41.962302 (+ 909us) server_negotiation.cc:299] Negotiation successful
0812 02:41:41.962663 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":136,"threads_started":1}
W20260812 02:41:41.965484 5416 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:41:41.965565 5416 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:41:41.965629 5401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:41.966477 5401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:41.967178 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:41:41.972546 5401 catalog_manager.cc:1347] Generated new cluster ID: 90b4821ce07b49bda7cc5414908a2ba9
I20260812 02:41:41.972632 5401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:41.978269 5380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:41.969662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36967 (local address 127.0.0.1:35129)
0812 02:41:41.969792 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:41.969797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:41.970145 (+ 348us) server_negotiation.cc:407] Connection header received
0812 02:41:41.970306 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:41.970310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:41.970360 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:41.970439 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:41.971342 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:41.972101 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:41.973091 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:41.973467 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:41.973574 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:41.974956 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:41.974960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:41.974963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:41.974993 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:41.976411 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:41.976918 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:41.976922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:41.976924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:41.977038 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:41.977381 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:41.977384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:41.977386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:41.977594 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:41.977728 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:41.977972 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 02:41:41.978105 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":46}
I20260812 02:41:41.987327 5401 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:41:41.988471 5401 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:42.003979 5401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87b1f20b6b8d4c59a36813e2c813a8e1: Generated new TSK 0
I20260812 02:41:42.004779 5401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:41:42.016114 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 5273
2026-08-12T02:41:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1604 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:41:42 dist-test-slave-rmhg krb5kdc[5429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:41:42 dist-test-slave-rmhg krb5kdc[5429](info): set up 2 sockets
Aug 12 02:41:42 dist-test-slave-rmhg krb5kdc[5429](info): commencing operation
krb5kdc: starting...
W20260812 02:41:43.195222 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:41:43 dist-test-slave-rmhg krb5kdc[5429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:41:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:41:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:41:43.330438 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44141
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:43.448184 5447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:43.448475 5447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:43.448518 5447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:43.452700 5447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:43.452791 5447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:43.452811 5447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:43.452826 5447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:43.452840 5447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:43.459364 5447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:43.460659 5447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:43.461839 5447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:43.471046 5456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:43.471046 5460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:43.471124 5447 server_base.cc:1034] running on GCE node
W20260812 02:41:43.471046 5457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:43.471935 5447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:43.473250 5447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:43.474422 5447 hybrid_clock.cc:648] HybridClock initialized: now 1786502503474395 us; error 50 us; skew 500 ppm
Aug 12 02:41:43 dist-test-slave-rmhg krb5kdc[5429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:43.477649 5447 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:41:43.478746 5447 webserver.cc:460] Webserver started at http://127.0.0.1:41347/ using document root <none> and password file <none>
I20260812 02:41:43.479347 5447 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:43.479413 5447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:43.479597 5447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:43.481885 5447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "739f75dd718f4831bbdbe92e0101e80f"
format_stamp: "Formatted at 2026-08-12 02:41:43 on dist-test-slave-rmhg"
server_key: "b4bcb9432460ddfc9d7415d210f3048c"
server_key_iv: "2ca82800e8a0f5d4a40731044ae86ef7"
server_key_version: "encryptionkey@0"
I20260812 02:41:43.482463 5447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "739f75dd718f4831bbdbe92e0101e80f"
format_stamp: "Formatted at 2026-08-12 02:41:43 on dist-test-slave-rmhg"
server_key: "b4bcb9432460ddfc9d7415d210f3048c"
server_key_iv: "2ca82800e8a0f5d4a40731044ae86ef7"
server_key_version: "encryptionkey@0"
I20260812 02:41:43.486519 5447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:41:43.489362 5474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:43.490758 5447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:41:43.490882 5447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "739f75dd718f4831bbdbe92e0101e80f"
format_stamp: "Formatted at 2026-08-12 02:41:43 on dist-test-slave-rmhg"
server_key: "b4bcb9432460ddfc9d7415d210f3048c"
server_key_iv: "2ca82800e8a0f5d4a40731044ae86ef7"
server_key_version: "encryptionkey@0"
I20260812 02:41:43.491014 5447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:43.505461 5447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:43.508731 5447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:43.508924 5447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:43.516212 5447 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44141
I20260812 02:41:43.516249 5560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44141 every 8 connection(s)
I20260812 02:41:43.517290 5447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:43.518472 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 5447
I20260812 02:41:43.518613 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:41:43.518879 3582 external_mini_cluster.cc:1428] Setting key 9e9693690e4af7d6b75e3ff83ad92ea6
I20260812 02:41:43.520607 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:43.527175 5562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f: Bootstrap starting.
I20260812 02:41:43.528524 5566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:43.520537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58159 (local address 127.0.0.1:44141)
0812 02:41:43.521014 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:43.521030 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:43.521068 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:41:43.521775 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:43.521792 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:43.522145 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:43.522429 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:43.522818 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:43.523822 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:43.525068 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:43.525604 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:43.525734 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:43.526627 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:43.526632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:43.526677 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:43.527077 (+ 400us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:43.527149 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:43.527173 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:43.527248 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:43.527435 (+ 187us) server_negotiation.cc:299] Negotiation successful
0812 02:41:43.527669 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":151,"threads_started":1}
I20260812 02:41:43.529881 5562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:43.530655 5562 log.cc:824] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:43.531859 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:41:43.533283 5562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f: No bootstrap required, opened a new log
I20260812 02:41:43.537937 5562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739f75dd718f4831bbdbe92e0101e80f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44141 } }
I20260812 02:41:43.538205 5562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:43.538267 5562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 739f75dd718f4831bbdbe92e0101e80f, State: Initialized, Role: FOLLOWER
I20260812 02:41:43.538741 5562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739f75dd718f4831bbdbe92e0101e80f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44141 } }
I20260812 02:41:43.538882 5562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:43.538946 5562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:43.539067 5562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:43.539072 5566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:43.534579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59831 (local address 127.0.0.1:44141)
0812 02:41:43.534698 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:43.534703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:43.534890 (+ 187us) server_negotiation.cc:407] Connection header received
0812 02:41:43.535057 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:43.535061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:43.535141 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:43.535225 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:43.535627 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:43.536412 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:43.537487 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:43.537747 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:43.537839 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:43.538629 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:43.538633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:43.538636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:43.538649 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:43.538676 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:43.538686 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:43.538743 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:43.538842 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 02:41:43.538897 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 02:41:43.540035 5562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739f75dd718f4831bbdbe92e0101e80f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44141 } }
I20260812 02:41:43.540388 5562 leader_election.cc:302] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 739f75dd718f4831bbdbe92e0101e80f; no voters:
I20260812 02:41:43.540710 5562 leader_election.cc:288] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:41:43.540838 5576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:43.541067 5576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f [term 1 LEADER]: Becoming Leader. State: Replica: 739f75dd718f4831bbdbe92e0101e80f, State: Running, Role: LEADER
I20260812 02:41:43.541388 5576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739f75dd718f4831bbdbe92e0101e80f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44141 } }
I20260812 02:41:43.542001 5562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:43.543140 5578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 739f75dd718f4831bbdbe92e0101e80f. Latest consensus state: current_term: 1 leader_uuid: "739f75dd718f4831bbdbe92e0101e80f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739f75dd718f4831bbdbe92e0101e80f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44141 } } }
I20260812 02:41:43.543140 5577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "739f75dd718f4831bbdbe92e0101e80f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739f75dd718f4831bbdbe92e0101e80f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44141 } } }
I20260812 02:41:43.543303 5578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:43.543311 5577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:43.543704 5592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:43.547143 5592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:43.553272 5592 catalog_manager.cc:1347] Generated new cluster ID: c5ce349e9d104a708fc161270c59f745
I20260812 02:41:43.553364 5592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:43.560351 5592 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:41:43.561892 5592 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:43.571516 5592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 739f75dd718f4831bbdbe92e0101e80f: Generated new TSK 0
I20260812 02:41:43.572314 5592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:41:43.576100 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 5447
2026-08-12T02:41:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1556 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:41:43 dist-test-slave-rmhg krb5kdc[5609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:41:43 dist-test-slave-rmhg krb5kdc[5609](info): set up 2 sockets
Aug 12 02:41:43 dist-test-slave-rmhg krb5kdc[5609](info): commencing operation
krb5kdc: starting...
W20260812 02:41:44.735903 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:41:44 dist-test-slave-rmhg krb5kdc[5609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:41:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:41:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:41:44.870299 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44129
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:44.988291 5632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:44.988569 5632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:44.988610 5632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:44.992642 5632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:44.992722 5632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:44.992740 5632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:44.992755 5632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:44.992769 5632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:44.998302 5632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:44.999387 5632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:45.000401 5632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:45.009382 5640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:45.009402 5642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:45.009459 5632 server_base.cc:1034] running on GCE node
W20260812 02:41:45.009421 5644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:45.010342 5632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:45.011579 5632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:45.012758 5632 hybrid_clock.cc:648] HybridClock initialized: now 1786502505012730 us; error 43 us; skew 500 ppm
Aug 12 02:41:45 dist-test-slave-rmhg krb5kdc[5609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:45.015630 5632 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:41:45.016619 5632 webserver.cc:460] Webserver started at http://127.0.0.1:42717/ using document root <none> and password file <none>
I20260812 02:41:45.017256 5632 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:45.017339 5632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:45.017570 5632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:45.019691 5632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "ad672c1294f942fb8f54bdf6026cf219"
format_stamp: "Formatted at 2026-08-12 02:41:45 on dist-test-slave-rmhg"
server_key: "62cd996ce6618e48b7210aea0ff83f76"
server_key_iv: "05446a3c5a0a193e983416045baa4c45"
server_key_version: "encryptionkey@0"
I20260812 02:41:45.020274 5632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "ad672c1294f942fb8f54bdf6026cf219"
format_stamp: "Formatted at 2026-08-12 02:41:45 on dist-test-slave-rmhg"
server_key: "62cd996ce6618e48b7210aea0ff83f76"
server_key_iv: "05446a3c5a0a193e983416045baa4c45"
server_key_version: "encryptionkey@0"
I20260812 02:41:45.024250 5632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:41:45.026823 5657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:45.028084 5632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 02:41:45.028240 5632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "ad672c1294f942fb8f54bdf6026cf219"
format_stamp: "Formatted at 2026-08-12 02:41:45 on dist-test-slave-rmhg"
server_key: "62cd996ce6618e48b7210aea0ff83f76"
server_key_iv: "05446a3c5a0a193e983416045baa4c45"
server_key_version: "encryptionkey@0"
I20260812 02:41:45.028353 5632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:45.033104 5632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:45.036033 5632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:45.036217 5632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:45.043435 5632 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44129
I20260812 02:41:45.043493 5741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44129 every 8 connection(s)
I20260812 02:41:45.044456 5632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:45.047554 5742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:45.048121 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 5632
I20260812 02:41:45.048290 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:41:45.048631 3582 external_mini_cluster.cc:1428] Setting key 48e7b346cc4ba4629d0b20c025d2155c
I20260812 02:41:45.054037 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:45.050140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50963 (local address 127.0.0.1:44129)
0812 02:41:45.050598 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:45.050614 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:45.050659 (+ 45us) server_negotiation.cc:407] Connection header received
0812 02:41:45.051355 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:45.051375 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:45.051687 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:45.052017 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:45.052822 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:45.052827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:45.052840 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:45.052851 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:45.052910 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:45.052929 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:45.053059 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:45.053133 (+ 74us) server_negotiation.cc:299] Negotiation successful
0812 02:41:45.053323 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":144,"threads_started":1}
I20260812 02:41:45.054854 5742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219: Bootstrap starting.
I20260812 02:41:45.056784 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:41:45.057335 5742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:45.058142 5742 log.cc:824] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:45.060313 5742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219: No bootstrap required, opened a new log
I20260812 02:41:45.062045 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:45.058856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35787 (local address 127.0.0.1:44129)
0812 02:41:45.058984 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:45.058988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:45.060717 (+ 1729us) server_negotiation.cc:407] Connection header received
0812 02:41:45.060836 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:45.060838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:45.060872 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:45.060939 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:45.061692 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:45.061695 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:45.061697 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:45.061708 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:45.061736 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:45.061746 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:45.061819 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:45.061901 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 02:41:45.061934 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 02:41:45.064527 5742 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad672c1294f942fb8f54bdf6026cf219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44129 } }
I20260812 02:41:45.064798 5742 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:45.064872 5742 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad672c1294f942fb8f54bdf6026cf219, State: Initialized, Role: FOLLOWER
I20260812 02:41:45.065371 5742 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad672c1294f942fb8f54bdf6026cf219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44129 } }
I20260812 02:41:45.065522 5742 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:45.065594 5742 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:45.065701 5742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:45.066658 5742 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad672c1294f942fb8f54bdf6026cf219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44129 } }
I20260812 02:41:45.067034 5742 leader_election.cc:302] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad672c1294f942fb8f54bdf6026cf219; no voters:
I20260812 02:41:45.067386 5742 leader_election.cc:288] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:41:45.067494 5764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:45.067729 5764 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219 [term 1 LEADER]: Becoming Leader. State: Replica: ad672c1294f942fb8f54bdf6026cf219, State: Running, Role: LEADER
I20260812 02:41:45.068064 5764 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad672c1294f942fb8f54bdf6026cf219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44129 } }
I20260812 02:41:45.068642 5742 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:45.069756 5772 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad672c1294f942fb8f54bdf6026cf219. Latest consensus state: current_term: 1 leader_uuid: "ad672c1294f942fb8f54bdf6026cf219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad672c1294f942fb8f54bdf6026cf219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44129 } } }
I20260812 02:41:45.069756 5771 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad672c1294f942fb8f54bdf6026cf219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad672c1294f942fb8f54bdf6026cf219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44129 } } }
I20260812 02:41:45.069967 5772 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:45.069974 5771 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:45.070452 5787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:45.073442 5787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:45.079329 5787 catalog_manager.cc:1347] Generated new cluster ID: a5f6bcbaa5d64f41adc055ae8aee2161
I20260812 02:41:45.079422 5787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:45.101651 5787 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:41:45.102088 5787 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:45.108163 5787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad672c1294f942fb8f54bdf6026cf219: Generated new TSK 0
I20260812 02:41:45.109123 5787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:41:45.121626 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 5632
2026-08-12T02:41:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1563 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:41:45 dist-test-slave-rmhg krb5kdc[5808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:41:45 dist-test-slave-rmhg krb5kdc[5808](info): set up 2 sockets
Aug 12 02:41:45 dist-test-slave-rmhg krb5kdc[5808](info): commencing operation
krb5kdc: starting...
W20260812 02:41:46.307085 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:41:46 dist-test-slave-rmhg krb5kdc[5808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:41:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:41:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:41:46.436836 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33893
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:46.556144 5827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:46.556460 5827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:46.556525 5827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:46.561144 5827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:46.561271 5827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:46.561297 5827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:46.561321 5827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:46.561350 5827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:46.567075 5827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:46.568287 5827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:46.569758 5827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:46.579192 5837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:46.579192 5842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:46.579303 5827 server_base.cc:1034] running on GCE node
W20260812 02:41:46.579252 5845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:46.580173 5827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:46.581439 5827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:46.582638 5827 hybrid_clock.cc:648] HybridClock initialized: now 1786502506582618 us; error 43 us; skew 500 ppm
Aug 12 02:41:46 dist-test-slave-rmhg krb5kdc[5808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:46.585824 5827 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:41:46.586912 5827 webserver.cc:460] Webserver started at http://127.0.0.1:41727/ using document root <none> and password file <none>
I20260812 02:41:46.587498 5827 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:46.587568 5827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:46.587774 5827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:46.590031 5827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "05f50b7046b64d46a8081a470109504d"
format_stamp: "Formatted at 2026-08-12 02:41:46 on dist-test-slave-rmhg"
server_key: "e800348840d9631ce3671f6ff58caf03"
server_key_iv: "4c41e4f798d12b167275938dfec15712"
server_key_version: "encryptionkey@0"
I20260812 02:41:46.590596 5827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "05f50b7046b64d46a8081a470109504d"
format_stamp: "Formatted at 2026-08-12 02:41:46 on dist-test-slave-rmhg"
server_key: "e800348840d9631ce3671f6ff58caf03"
server_key_iv: "4c41e4f798d12b167275938dfec15712"
server_key_version: "encryptionkey@0"
I20260812 02:41:46.594501 5827 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:41:46.596972 5855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:46.598433 5827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:41:46.598559 5827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "05f50b7046b64d46a8081a470109504d"
format_stamp: "Formatted at 2026-08-12 02:41:46 on dist-test-slave-rmhg"
server_key: "e800348840d9631ce3671f6ff58caf03"
server_key_iv: "4c41e4f798d12b167275938dfec15712"
server_key_version: "encryptionkey@0"
I20260812 02:41:46.598670 5827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:46.625442 5827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:46.628328 5827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:46.628520 5827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:46.635846 5827 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33893
I20260812 02:41:46.635876 5939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33893 every 8 connection(s)
I20260812 02:41:46.636864 5827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:46.639802 5940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:46.645182 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 5827
I20260812 02:41:46.645325 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:41:46.645615 3582 external_mini_cluster.cc:1428] Setting key c22a1ea26af34936c94d3545dfa68529
I20260812 02:41:46.645999 5940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d: Bootstrap starting.
I20260812 02:41:46.648738 5940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:46.649705 5940 log.cc:824] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:46.651896 5940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d: No bootstrap required, opened a new log
Aug 12 02:41:46 dist-test-slave-rmhg krb5kdc[5808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 02:41:46.655827 5940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f50b7046b64d46a8081a470109504d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33893 } }
I20260812 02:41:46.656111 5940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:46.656179 5940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f50b7046b64d46a8081a470109504d, State: Initialized, Role: FOLLOWER
I20260812 02:41:46.656733 5940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f50b7046b64d46a8081a470109504d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33893 } }
I20260812 02:41:46.656891 5940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:46.656963 5940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:46.657073 5940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:46.658030 5940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f50b7046b64d46a8081a470109504d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33893 } }
I20260812 02:41:46.658382 5940 leader_election.cc:302] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05f50b7046b64d46a8081a470109504d; no voters:
I20260812 02:41:46.658783 5940 leader_election.cc:288] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:41:46.658883 5947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:46.659104 5947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d [term 1 LEADER]: Becoming Leader. State: Replica: 05f50b7046b64d46a8081a470109504d, State: Running, Role: LEADER
I20260812 02:41:46.659488 5947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f50b7046b64d46a8081a470109504d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33893 } }
I20260812 02:41:46.660269 5940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:46.661227 5949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05f50b7046b64d46a8081a470109504d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f50b7046b64d46a8081a470109504d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33893 } } }
I20260812 02:41:46.661367 5949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:46.661707 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:46.647328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:43435 (local address 127.0.0.1:33893)
0812 02:41:46.647804 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:46.647822 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:46.647862 (+ 40us) server_negotiation.cc:407] Connection header received
0812 02:41:46.648561 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:46.648581 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:46.648998 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:46.649379 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:46.650353 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:46.651506 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:46.652433 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:46.652938 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:46.653097 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:46.655689 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:46.655694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:46.655709 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:46.655739 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:46.657609 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:46.658144 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:46.658150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:46.658156 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:46.658284 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:46.658640 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:46.658643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:46.658645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:46.659223 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:46.660100 (+ 877us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:46.660289 (+ 189us) server_negotiation.cc:299] Negotiation successful
0812 02:41:46.660551 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":141,"threads_started":1}
I20260812 02:41:46.661718 5953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05f50b7046b64d46a8081a470109504d. Latest consensus state: current_term: 1 leader_uuid: "05f50b7046b64d46a8081a470109504d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f50b7046b64d46a8081a470109504d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33893 } } }
I20260812 02:41:46.662063 5953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:46.665684 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260812 02:41:46.665706 5972 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:41:46.665768 5972 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:41:46.665827 5965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:46.666496 5965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:46.676882 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:46.668023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:59773 (local address 127.0.0.1:33893)
0812 02:41:46.668135 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:46.668139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:46.669513 (+ 1374us) server_negotiation.cc:407] Connection header received
0812 02:41:46.669628 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:46.669632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:46.669690 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:46.669783 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:46.670538 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:46.671064 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:46.671870 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:46.672124 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:46.672216 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:46.673783 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:46.673787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:46.673789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:46.673815 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:46.675111 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:46.675589 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:46.675592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:46.675593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:46.675681 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:46.676010 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:46.676013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:46.676014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:46.676203 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:46.676334 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:46.676613 (+ 279us) server_negotiation.cc:299] Negotiation successful
0812 02:41:46.676726 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 02:41:46.679345 5965 catalog_manager.cc:1347] Generated new cluster ID: d2fec69bb6394851befeda64de7e1d29
I20260812 02:41:46.679418 5965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:46.698038 5965 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:41:46.699100 5965 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:46.708611 5965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05f50b7046b64d46a8081a470109504d: Generated new TSK 0
I20260812 02:41:46.709473 5965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:41:46.714279 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 5827
W20260812 02:41:46.728912 5803 connection.cc:535] client connection to 127.0.0.1:33893 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33893): Connection reset by peer (error 104)
2026-08-12T02:41:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1577 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:41:46 dist-test-slave-rmhg krb5kdc[5987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:41:46 dist-test-slave-rmhg krb5kdc[5987](info): set up 2 sockets
Aug 12 02:41:46 dist-test-slave-rmhg krb5kdc[5987](info): commencing operation
krb5kdc: starting...
W20260812 02:41:47.886379 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:41:47 dist-test-slave-rmhg krb5kdc[5987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:41:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:41:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:41:48.016106 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:48.134658 6004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:48.134948 6004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:48.135011 6004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:48.139109 6004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:48.139200 6004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:48.139230 6004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:48.139250 6004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:48.139273 6004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:48.144968 6004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:48.146145 6004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:48.147192 6004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:41:48.156433 6004 server_base.cc:1034] running on GCE node
W20260812 02:41:48.156389 6011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:48.156368 6012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:48.156360 6016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:48.157380 6004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:48.158627 6004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:48.159826 6004 hybrid_clock.cc:648] HybridClock initialized: now 1786502508159807 us; error 38 us; skew 500 ppm
Aug 12 02:41:48 dist-test-slave-rmhg krb5kdc[5987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:48.166541 6004 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:41:48.167673 6004 webserver.cc:460] Webserver started at http://127.0.0.1:44545/ using document root <none> and password file <none>
I20260812 02:41:48.168287 6004 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:48.168368 6004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:48.168596 6004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:48.170876 6004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "934617a8e3424b0e89c2f39be597d4d6"
format_stamp: "Formatted at 2026-08-12 02:41:48 on dist-test-slave-rmhg"
server_key: "58fde7d30879b138d5d6ac5c7e688bd5"
server_key_iv: "695b3da11574f333e489fb4f183853f7"
server_key_version: "encryptionkey@0"
I20260812 02:41:48.171468 6004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "934617a8e3424b0e89c2f39be597d4d6"
format_stamp: "Formatted at 2026-08-12 02:41:48 on dist-test-slave-rmhg"
server_key: "58fde7d30879b138d5d6ac5c7e688bd5"
server_key_iv: "695b3da11574f333e489fb4f183853f7"
server_key_version: "encryptionkey@0"
I20260812 02:41:48.175613 6004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260812 02:41:48.178337 6027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:48.179641 6004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:41:48.179812 6004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "934617a8e3424b0e89c2f39be597d4d6"
format_stamp: "Formatted at 2026-08-12 02:41:48 on dist-test-slave-rmhg"
server_key: "58fde7d30879b138d5d6ac5c7e688bd5"
server_key_iv: "695b3da11574f333e489fb4f183853f7"
server_key_version: "encryptionkey@0"
I20260812 02:41:48.179934 6004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:48.206194 6004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:48.209613 6004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:48.209820 6004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:48.217051 6004 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38315
I20260812 02:41:48.217141 6112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38315 every 8 connection(s)
I20260812 02:41:48.218154 6004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:48.221300 6113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:48.224246 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 6004
I20260812 02:41:48.224377 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:41:48.224642 3582 external_mini_cluster.cc:1428] Setting key 72d7cdf922539b12fffc86765442a1ff
I20260812 02:41:48.227573 6113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6: Bootstrap starting.
I20260812 02:41:48.230441 6113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:48.231251 6113 log.cc:824] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:48.233780 6113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6: No bootstrap required, opened a new log
I20260812 02:41:48.234293 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:48.226059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:43079 (local address 127.0.0.1:38315)
0812 02:41:48.226505 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:48.226522 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:48.226559 (+ 37us) server_negotiation.cc:407] Connection header received
0812 02:41:48.227194 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:48.227212 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:48.227577 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:48.227895 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:48.228263 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:48.229417 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:48.230571 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:48.231080 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:48.231234 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:48.232171 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:48.232178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:48.232196 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:48.232703 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:48.232777 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:48.232809 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:48.232957 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:48.233132 (+ 175us) server_negotiation.cc:299] Negotiation successful
0812 02:41:48.233305 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":131,"threads_started":1}
I20260812 02:41:48.237543 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:41:48.238093 6113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "934617a8e3424b0e89c2f39be597d4d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38315 } }
I20260812 02:41:48.238397 6113 raft_consensus.cc:374] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:48.238473 6113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 934617a8e3424b0e89c2f39be597d4d6, State: Initialized, Role: FOLLOWER
I20260812 02:41:48.238996 6113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "934617a8e3424b0e89c2f39be597d4d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38315 } }
I20260812 02:41:48.239179 6113 raft_consensus.cc:388] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:48.239254 6113 raft_consensus.cc:482] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:48.239360 6113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:48.240563 6113 raft_consensus.cc:504] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "934617a8e3424b0e89c2f39be597d4d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38315 } }
I20260812 02:41:48.240991 6113 leader_election.cc:302] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 934617a8e3424b0e89c2f39be597d4d6; no voters:
I20260812 02:41:48.241385 6113 leader_election.cc:288] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:41:48.241487 6127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:48.241824 6127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6 [term 1 LEADER]: Becoming Leader. State: Replica: 934617a8e3424b0e89c2f39be597d4d6, State: Running, Role: LEADER
I20260812 02:41:48.242192 6127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "934617a8e3424b0e89c2f39be597d4d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38315 } }
I20260812 02:41:48.242749 6113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:48.243532 6127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 934617a8e3424b0e89c2f39be597d4d6. Latest consensus state: current_term: 1 leader_uuid: "934617a8e3424b0e89c2f39be597d4d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "934617a8e3424b0e89c2f39be597d4d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38315 } } }
I20260812 02:41:48.243667 6127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:48.244324 6128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "934617a8e3424b0e89c2f39be597d4d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "934617a8e3424b0e89c2f39be597d4d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38315 } } }
I20260812 02:41:48.244504 6128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:48.244838 6133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:48.245010 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:48.239906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:57791 (local address 127.0.0.1:38315)
0812 02:41:48.240029 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:48.240033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:48.240220 (+ 187us) server_negotiation.cc:407] Connection header received
0812 02:41:48.240310 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:48.240313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:48.240368 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:48.240450 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:48.240679 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:48.241488 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:48.242658 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:48.242995 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:48.243100 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:48.244127 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:48.244132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:48.244134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:48.244162 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:48.244192 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:48.244206 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:48.244296 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:48.244816 (+ 520us) server_negotiation.cc:299] Negotiation successful
0812 02:41:48.244861 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 02:41:48.247934 6133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:48.253769 6133 catalog_manager.cc:1347] Generated new cluster ID: 648c7f8e227d497e8ef36e9ac7176a30
I20260812 02:41:48.253849 6133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:48.269374 6133 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:41:48.270594 6133 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:48.281958 6133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 934617a8e3424b0e89c2f39be597d4d6: Generated new TSK 0
I20260812 02:41:48.282968 6133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:41:48.294256 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 6004
W20260812 02:41:48.312204 5984 connection.cc:535] client connection to 127.0.0.1:38315 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38315): Connection reset by peer (error 104)
2026-08-12T02:41:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1583 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:41:48 dist-test-slave-rmhg krb5kdc[6155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:41:48 dist-test-slave-rmhg krb5kdc[6155](info): set up 2 sockets
Aug 12 02:41:48 dist-test-slave-rmhg krb5kdc[6155](info): commencing operation
krb5kdc: starting...
W20260812 02:41:49.478536 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:41:49 dist-test-slave-rmhg krb5kdc[6155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:41:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:41:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:41:49.611829 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33493
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:49.728580 6177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:49.728858 6177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:49.728904 6177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:49.732977 6177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:49.733045 6177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:49.733071 6177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:49.733088 6177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:49.733103 6177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:49.738646 6177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:49.739744 6177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:49.740785 6177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:49.749413 6188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:49.749480 6177 server_base.cc:1034] running on GCE node
W20260812 02:41:49.749450 6183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:49.749426 6184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:49.750228 6177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:49.751374 6177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:49.752566 6177 hybrid_clock.cc:648] HybridClock initialized: now 1786502509752545 us; error 34 us; skew 500 ppm
Aug 12 02:41:49 dist-test-slave-rmhg krb5kdc[6155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:49.755465 6177 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:41:49.756415 6177 webserver.cc:460] Webserver started at http://127.0.0.1:42035/ using document root <none> and password file <none>
I20260812 02:41:49.757107 6177 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:49.757222 6177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:49.757452 6177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:49.759618 6177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "0e08cce7aec34817b58799bddda7657b"
format_stamp: "Formatted at 2026-08-12 02:41:49 on dist-test-slave-rmhg"
server_key: "8beb9c01af4275c9e08566c2c186c7fb"
server_key_iv: "66463d23a1125e7438590dbe09c250e4"
server_key_version: "encryptionkey@0"
I20260812 02:41:49.760249 6177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "0e08cce7aec34817b58799bddda7657b"
format_stamp: "Formatted at 2026-08-12 02:41:49 on dist-test-slave-rmhg"
server_key: "8beb9c01af4275c9e08566c2c186c7fb"
server_key_iv: "66463d23a1125e7438590dbe09c250e4"
server_key_version: "encryptionkey@0"
I20260812 02:41:49.764418 6177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 02:41:49.767175 6198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:49.768469 6177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 02:41:49.768590 6177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "0e08cce7aec34817b58799bddda7657b"
format_stamp: "Formatted at 2026-08-12 02:41:49 on dist-test-slave-rmhg"
server_key: "8beb9c01af4275c9e08566c2c186c7fb"
server_key_iv: "66463d23a1125e7438590dbe09c250e4"
server_key_version: "encryptionkey@0"
I20260812 02:41:49.768718 6177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:49.773950 6177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:49.777257 6177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:49.777446 6177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:49.784716 6177 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33493
I20260812 02:41:49.784746 6284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33493 every 8 connection(s)
I20260812 02:41:49.785779 6177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:49.788694 6285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:49.789773 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 6177
I20260812 02:41:49.789913 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:41:49.790248 3582 external_mini_cluster.cc:1428] Setting key a1c1b62b85685fe3caaf4ce8ebacedd1
I20260812 02:41:49.795269 6285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b: Bootstrap starting.
I20260812 02:41:49.795611 6290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:49.791697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:55451 (local address 127.0.0.1:33493)
0812 02:41:49.792125 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:49.792141 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:49.792190 (+ 49us) server_negotiation.cc:407] Connection header received
0812 02:41:49.792820 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:49.792837 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:49.793132 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:49.793470 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:49.794307 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:49.794312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:49.794324 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:49.794334 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:49.794391 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:49.794410 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:49.794519 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:49.794642 (+ 123us) server_negotiation.cc:299] Negotiation successful
0812 02:41:49.794811 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":129,"threads_started":1}
I20260812 02:41:49.797842 6285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:49.798569 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:41:49.798679 6285 log.cc:824] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:49.801057 6285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b: No bootstrap required, opened a new log
I20260812 02:41:49.802436 6290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:49.800840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:48555 (local address 127.0.0.1:33493)
0812 02:41:49.800966 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:49.800970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:49.801176 (+ 206us) server_negotiation.cc:407] Connection header received
0812 02:41:49.801307 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:49.801310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:49.801348 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:49.801434 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:49.802067 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:49.802071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:49.802073 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:49.802087 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:49.802117 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:49.802130 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:49.802211 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:49.802267 (+ 56us) server_negotiation.cc:299] Negotiation successful
0812 02:41:49.802306 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 02:41:49.805022 6285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e08cce7aec34817b58799bddda7657b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33493 } }
I20260812 02:41:49.805311 6285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:49.805382 6285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e08cce7aec34817b58799bddda7657b, State: Initialized, Role: FOLLOWER
I20260812 02:41:49.805846 6285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e08cce7aec34817b58799bddda7657b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33493 } }
I20260812 02:41:49.805984 6285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:49.806051 6285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:49.806162 6285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:49.807119 6285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e08cce7aec34817b58799bddda7657b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33493 } }
I20260812 02:41:49.807466 6285 leader_election.cc:302] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e08cce7aec34817b58799bddda7657b; no voters:
I20260812 02:41:49.807781 6285 leader_election.cc:288] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:41:49.807915 6301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:49.808161 6301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b [term 1 LEADER]: Becoming Leader. State: Replica: 0e08cce7aec34817b58799bddda7657b, State: Running, Role: LEADER
I20260812 02:41:49.808511 6301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e08cce7aec34817b58799bddda7657b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33493 } }
I20260812 02:41:49.809118 6285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:49.809722 6301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e08cce7aec34817b58799bddda7657b. Latest consensus state: current_term: 1 leader_uuid: "0e08cce7aec34817b58799bddda7657b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e08cce7aec34817b58799bddda7657b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33493 } } }
I20260812 02:41:49.809844 6301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:49.810169 6303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e08cce7aec34817b58799bddda7657b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e08cce7aec34817b58799bddda7657b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33493 } } }
I20260812 02:41:49.810316 6303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:49.810408 6312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:49.814011 6312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:49.819969 6312 catalog_manager.cc:1347] Generated new cluster ID: f4b4b86bad1945d4b884bde16b044586
I20260812 02:41:49.820058 6312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:49.827647 6312 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:41:49.828028 6312 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:49.844111 6312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e08cce7aec34817b58799bddda7657b: Generated new TSK 0
I20260812 02:41:49.844946 6312 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:41:49.850045 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 6177
2026-08-12T02:41:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1551 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:41:49 dist-test-slave-rmhg krb5kdc[6333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:41:49 dist-test-slave-rmhg krb5kdc[6333](info): set up 2 sockets
Aug 12 02:41:49 dist-test-slave-rmhg krb5kdc[6333](info): commencing operation
krb5kdc: starting...
W20260812 02:41:51.019074 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:41:51 dist-test-slave-rmhg krb5kdc[6333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:41:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:41:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:41:51.152280 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41559
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:51.272766 6352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:51.273113 6352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:51.273203 6352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:51.277500 6352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:51.277592 6352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:51.277634 6352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:51.277667 6352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:51.277697 6352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:51.283612 6352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41559
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:51.284873 6352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:51.286026 6352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:51.295517 6357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:51.295495 6361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:51.295560 6358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:51.295955 6352 server_base.cc:1034] running on GCE node
I20260812 02:41:51.296617 6352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:51.297989 6352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:51.299212 6352 hybrid_clock.cc:648] HybridClock initialized: now 1786502511299190 us; error 41 us; skew 500 ppm
Aug 12 02:41:51 dist-test-slave-rmhg krb5kdc[6333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:51.302495 6352 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:41:51.303661 6352 webserver.cc:460] Webserver started at http://127.0.0.1:44073/ using document root <none> and password file <none>
I20260812 02:41:51.304279 6352 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:51.304350 6352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:51.304577 6352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:51.306931 6352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "84d69570532d485c9ac6b3a67bb0bc42"
format_stamp: "Formatted at 2026-08-12 02:41:51 on dist-test-slave-rmhg"
server_key: "2fffbc01d8aaf684354771dbf4fa0423"
server_key_iv: "877b680f0da2da6e4597e496a87cb58d"
server_key_version: "encryptionkey@0"
I20260812 02:41:51.307595 6352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "84d69570532d485c9ac6b3a67bb0bc42"
format_stamp: "Formatted at 2026-08-12 02:41:51 on dist-test-slave-rmhg"
server_key: "2fffbc01d8aaf684354771dbf4fa0423"
server_key_iv: "877b680f0da2da6e4597e496a87cb58d"
server_key_version: "encryptionkey@0"
I20260812 02:41:51.312248 6352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260812 02:41:51.315214 6375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:51.316699 6352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:41:51.316859 6352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "84d69570532d485c9ac6b3a67bb0bc42"
format_stamp: "Formatted at 2026-08-12 02:41:51 on dist-test-slave-rmhg"
server_key: "2fffbc01d8aaf684354771dbf4fa0423"
server_key_iv: "877b680f0da2da6e4597e496a87cb58d"
server_key_version: "encryptionkey@0"
I20260812 02:41:51.316991 6352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:51.345985 6352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:51.349726 6352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:51.349939 6352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:51.357929 6352 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41559
I20260812 02:41:51.357954 6469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41559 every 8 connection(s)
I20260812 02:41:51.359062 6352 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:51.361126 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 6352
I20260812 02:41:51.361300 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:41:51.361594 3582 external_mini_cluster.cc:1428] Setting key 05d5962bf280dcae1f6d5bf1ded02e09
I20260812 02:41:51.362361 6470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:51.368970 6470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42: Bootstrap starting.
Aug 12 02:41:51 dist-test-slave-rmhg krb5kdc[6333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 02:41:51.371637 6470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:51.372534 6470 log.cc:824] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:51.374708 6470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42: No bootstrap required, opened a new log
I20260812 02:41:51.376587 6475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:51.363054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:55249 (local address 127.0.0.1:41559)
0812 02:41:51.363617 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:51.363640 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:51.363846 (+ 206us) server_negotiation.cc:407] Connection header received
0812 02:41:51.364662 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:51.364683 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:51.365061 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:51.365436 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:51.366288 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:51.367339 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:51.368226 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:51.368748 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:51.368906 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:51.371650 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:51.371656 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:51.371672 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:51.371708 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:51.373422 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:51.374014 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:51.374020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:51.374025 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:51.374149 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:51.374484 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:51.374487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:51.374488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:51.374952 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:51.375159 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:51.375512 (+ 353us) server_negotiation.cc:299] Negotiation successful
0812 02:41:51.375784 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":189,"threads_started":1}
I20260812 02:41:51.378914 6470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d69570532d485c9ac6b3a67bb0bc42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41559 } }
I20260812 02:41:51.379169 6470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:51.379240 6470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d69570532d485c9ac6b3a67bb0bc42, State: Initialized, Role: FOLLOWER
I20260812 02:41:51.379729 6470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d69570532d485c9ac6b3a67bb0bc42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41559 } }
I20260812 02:41:51.379908 6470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:51.379982 6470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:51.380089 6470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:51.380111 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:41:51.381094 6470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d69570532d485c9ac6b3a67bb0bc42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41559 } }
I20260812 02:41:51.381531 6470 leader_election.cc:302] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84d69570532d485c9ac6b3a67bb0bc42; no voters:
I20260812 02:41:51.381877 6470 leader_election.cc:288] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:41:51.381971 6483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:51.382206 6483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42 [term 1 LEADER]: Becoming Leader. State: Replica: 84d69570532d485c9ac6b3a67bb0bc42, State: Running, Role: LEADER
I20260812 02:41:51.382609 6483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d69570532d485c9ac6b3a67bb0bc42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41559 } }
I20260812 02:41:51.383571 6470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:51.384455 6487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84d69570532d485c9ac6b3a67bb0bc42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d69570532d485c9ac6b3a67bb0bc42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41559 } } }
I20260812 02:41:51.384608 6487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:51.384902 6484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84d69570532d485c9ac6b3a67bb0bc42. Latest consensus state: current_term: 1 leader_uuid: "84d69570532d485c9ac6b3a67bb0bc42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d69570532d485c9ac6b3a67bb0bc42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41559 } } }
I20260812 02:41:51.385002 6484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42 [sys.catalog]: This master's current role is: LEADER
W20260812 02:41:51.389461 6516 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:41:51.389556 6516 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:41:51.389650 6504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:51.390550 6504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:51.392287 6475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:51.382213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:43971 (local address 127.0.0.1:41559)
0812 02:41:51.382337 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:51.382341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:51.382779 (+ 438us) server_negotiation.cc:407] Connection header received
0812 02:41:51.382964 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:51.382967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:51.383023 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:51.383093 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:51.384095 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:51.384941 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:51.386079 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:51.386399 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:51.386474 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:51.388264 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:51.388268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:51.388271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:51.388300 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:51.390103 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:51.390803 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:51.390807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:51.390809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:51.390905 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:51.391310 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:51.391314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:51.391316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:51.391558 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:51.391715 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:51.392014 (+ 299us) server_negotiation.cc:299] Negotiation successful
0812 02:41:51.392119 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":48}
I20260812 02:41:51.397070 6504 catalog_manager.cc:1347] Generated new cluster ID: 95ccf57cc15a46bb944a63acd3f565fc
I20260812 02:41:51.397141 6504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:51.434088 6504 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:41:51.435149 6504 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:51.451416 6504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84d69570532d485c9ac6b3a67bb0bc42: Generated new TSK 0
I20260812 02:41:51.452211 6504 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:41:51.465299 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 6352
W20260812 02:41:51.479650 6330 connection.cc:535] client connection to 127.0.0.1:41559 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41559): Connection reset by peer (error 104)
2026-08-12T02:41:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1615 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:41:51 dist-test-slave-rmhg krb5kdc[6523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:41:51 dist-test-slave-rmhg krb5kdc[6523](info): set up 2 sockets
Aug 12 02:41:51 dist-test-slave-rmhg krb5kdc[6523](info): commencing operation
krb5kdc: starting...
W20260812 02:41:52.657696 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:41:52 dist-test-slave-rmhg krb5kdc[6523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:41:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:41:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:41:52.788338 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:52.904950 6549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:52.905303 6549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:52.905381 6549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:52.909402 6549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:52.909466 6549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:52.909492 6549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:52.909513 6549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:52.909534 6549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:52.914937 6549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39867
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:52.916054 6549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:52.917097 6549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:52.925768 6566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:52.925809 6549 server_base.cc:1034] running on GCE node
W20260812 02:41:52.925768 6559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:52.925792 6557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:52.926667 6549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:52.927780 6549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:52.928978 6549 hybrid_clock.cc:648] HybridClock initialized: now 1786502512928963 us; error 37 us; skew 500 ppm
Aug 12 02:41:52 dist-test-slave-rmhg krb5kdc[6523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:52.931774 6549 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:41:52.932710 6549 webserver.cc:460] Webserver started at http://127.0.0.1:40727/ using document root <none> and password file <none>
I20260812 02:41:52.933301 6549 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:52.933377 6549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:52.933589 6549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:52.935760 6549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "aed0ac172db646dbb1ed67901e7ae3f6"
format_stamp: "Formatted at 2026-08-12 02:41:52 on dist-test-slave-rmhg"
server_key: "3a19e7928b9990c98fadd9295596135f"
server_key_iv: "0297871065fd22c5f500bedea11eab68"
server_key_version: "encryptionkey@0"
I20260812 02:41:52.936326 6549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "aed0ac172db646dbb1ed67901e7ae3f6"
format_stamp: "Formatted at 2026-08-12 02:41:52 on dist-test-slave-rmhg"
server_key: "3a19e7928b9990c98fadd9295596135f"
server_key_iv: "0297871065fd22c5f500bedea11eab68"
server_key_version: "encryptionkey@0"
I20260812 02:41:52.940080 6549 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 02:41:52.942600 6576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:52.943847 6549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 02:41:52.943966 6549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "aed0ac172db646dbb1ed67901e7ae3f6"
format_stamp: "Formatted at 2026-08-12 02:41:52 on dist-test-slave-rmhg"
server_key: "3a19e7928b9990c98fadd9295596135f"
server_key_iv: "0297871065fd22c5f500bedea11eab68"
server_key_version: "encryptionkey@0"
I20260812 02:41:52.944063 6549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:52.960333 6549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:52.963511 6549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:52.963713 6549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:52.971028 6549 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39867
I20260812 02:41:52.971050 6663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39867 every 8 connection(s)
I20260812 02:41:52.972016 6549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:52.974970 6665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:52.976219 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 6549
I20260812 02:41:52.976356 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:41:52.976675 3582 external_mini_cluster.cc:1428] Setting key 1033cdb8a1b3bae3a587f3037fbc3975
I20260812 02:41:52.982260 6665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6: Bootstrap starting.
I20260812 02:41:52.984988 6665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:52.985983 6665 log.cc:824] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:52.987179 6674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:52.978210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:43035 (local address 127.0.0.1:39867)
0812 02:41:52.978681 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:52.978708 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:52.978763 (+ 55us) server_negotiation.cc:407] Connection header received
0812 02:41:52.979591 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:52.979618 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:52.980094 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:52.980498 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:52.980980 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:52.982504 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:52.983438 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:52.984165 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:52.984343 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:52.985139 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:52.985147 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:52.985192 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:52.985719 (+ 527us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:52.985802 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:52.985835 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:52.985972 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:52.986141 (+ 169us) server_negotiation.cc:299] Negotiation successful
0812 02:41:52.986283 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":106,"threads_started":1}
I20260812 02:41:52.988257 6665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6: No bootstrap required, opened a new log
I20260812 02:41:52.990371 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:41:52.992841 6665 raft_consensus.cc:348] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed0ac172db646dbb1ed67901e7ae3f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39867 } }
I20260812 02:41:52.993104 6665 raft_consensus.cc:374] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:52.993216 6665 raft_consensus.cc:729] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aed0ac172db646dbb1ed67901e7ae3f6, State: Initialized, Role: FOLLOWER
I20260812 02:41:52.993728 6665 consensus_queue.cc:260] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed0ac172db646dbb1ed67901e7ae3f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39867 } }
I20260812 02:41:52.993912 6665 raft_consensus.cc:388] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:52.993984 6665 raft_consensus.cc:482] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:52.994107 6665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:52.995169 6665 raft_consensus.cc:504] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed0ac172db646dbb1ed67901e7ae3f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39867 } }
I20260812 02:41:52.995585 6665 leader_election.cc:302] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aed0ac172db646dbb1ed67901e7ae3f6; no voters:
I20260812 02:41:52.995992 6665 leader_election.cc:288] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:41:52.996076 6686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:52.996295 6686 raft_consensus.cc:686] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6 [term 1 LEADER]: Becoming Leader. State: Replica: aed0ac172db646dbb1ed67901e7ae3f6, State: Running, Role: LEADER
I20260812 02:41:52.996632 6686 consensus_queue.cc:237] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed0ac172db646dbb1ed67901e7ae3f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39867 } }
I20260812 02:41:52.997071 6674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:52.992753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:40679 (local address 127.0.0.1:39867)
0812 02:41:52.992880 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:52.992884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:52.993144 (+ 260us) server_negotiation.cc:407] Connection header received
0812 02:41:52.993256 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:52.993259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:52.993318 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:52.993398 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:52.993704 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:52.994447 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:52.995262 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:52.995620 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:52.995733 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:52.996578 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:52.996581 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:52.996584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:52.996599 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:52.996633 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:52.996646 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:52.996766 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:52.996900 (+ 134us) server_negotiation.cc:299] Negotiation successful
0812 02:41:52.996936 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 02:41:52.997526 6665 sys_catalog.cc:564] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:52.999125 6687 sys_catalog.cc:455] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aed0ac172db646dbb1ed67901e7ae3f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed0ac172db646dbb1ed67901e7ae3f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39867 } } }
I20260812 02:41:52.999168 6689 sys_catalog.cc:455] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aed0ac172db646dbb1ed67901e7ae3f6. Latest consensus state: current_term: 1 leader_uuid: "aed0ac172db646dbb1ed67901e7ae3f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed0ac172db646dbb1ed67901e7ae3f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39867 } } }
I20260812 02:41:52.999272 6687 sys_catalog.cc:458] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:52.999279 6689 sys_catalog.cc:458] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6 [sys.catalog]: This master's current role is: LEADER
W20260812 02:41:53.003064 6714 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:41:53.003176 6714 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:41:53.003266 6711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:53.004177 6711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:53.010602 6711 catalog_manager.cc:1347] Generated new cluster ID: cdfedbd64b7145be83999ce35a1ccedb
I20260812 02:41:53.010676 6711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:53.022253 6711 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:41:53.023306 6711 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:53.031525 6711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aed0ac172db646dbb1ed67901e7ae3f6: Generated new TSK 0
I20260812 02:41:53.032307 6711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:41:53.042140 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 6549
W20260812 02:41:53.054208 6520 connection.cc:535] client connection to 127.0.0.1:39867 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39867): Connection reset by peer (error 104)
2026-08-12T02:41:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1574 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:41:53 dist-test-slave-rmhg krb5kdc[6722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:41:53 dist-test-slave-rmhg krb5kdc[6722](info): set up 2 sockets
Aug 12 02:41:53 dist-test-slave-rmhg krb5kdc[6722](info): commencing operation
krb5kdc: starting...
W20260812 02:41:54.208487 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:41:54 dist-test-slave-rmhg krb5kdc[6722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:41:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:41:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:41:54.341681 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41907
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:54.461006 6751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:54.461367 6751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:54.461417 6751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:54.465507 6751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:54.465574 6751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:54.465591 6751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:54.465607 6751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:54.465627 6751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:54.471201 6751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41907
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:54.472291 6751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:54.473335 6751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:54.482035 6760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:54.482008 6762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:54.482185 6751 server_base.cc:1034] running on GCE node
W20260812 02:41:54.482028 6759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:54.482856 6751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:54.483999 6751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:54.485219 6751 hybrid_clock.cc:648] HybridClock initialized: now 1786502514485188 us; error 41 us; skew 500 ppm
Aug 12 02:41:54 dist-test-slave-rmhg krb5kdc[6722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:54.491549 6751 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:41:54.492551 6751 webserver.cc:460] Webserver started at http://127.0.0.1:43203/ using document root <none> and password file <none>
I20260812 02:41:54.493096 6751 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:54.493181 6751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:54.493388 6751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:54.495435 6751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "dec68b27f5ca486e95edbd1c4ae56ac4"
format_stamp: "Formatted at 2026-08-12 02:41:54 on dist-test-slave-rmhg"
server_key: "186978a61629c14e50b51107cc144551"
server_key_iv: "c2bfcee99864acf6866b6feefc838851"
server_key_version: "encryptionkey@0"
I20260812 02:41:54.495950 6751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "dec68b27f5ca486e95edbd1c4ae56ac4"
format_stamp: "Formatted at 2026-08-12 02:41:54 on dist-test-slave-rmhg"
server_key: "186978a61629c14e50b51107cc144551"
server_key_iv: "c2bfcee99864acf6866b6feefc838851"
server_key_version: "encryptionkey@0"
I20260812 02:41:54.499832 6751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:41:54.502467 6776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:54.503858 6751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:41:54.504022 6751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "dec68b27f5ca486e95edbd1c4ae56ac4"
format_stamp: "Formatted at 2026-08-12 02:41:54 on dist-test-slave-rmhg"
server_key: "186978a61629c14e50b51107cc144551"
server_key_iv: "c2bfcee99864acf6866b6feefc838851"
server_key_version: "encryptionkey@0"
I20260812 02:41:54.504134 6751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:54.509039 6751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:54.512629 6751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:54.512843 6751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:54.520093 6751 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41907
I20260812 02:41:54.520133 6864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41907 every 8 connection(s)
I20260812 02:41:54.521076 6751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:54.524098 6865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:54.529664 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 6751
I20260812 02:41:54.529815 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:41:54.530021 6865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4: Bootstrap starting.
I20260812 02:41:54.530107 3582 external_mini_cluster.cc:1428] Setting key 3243528c3c03eb647a9f3b2de63e6f7b
I20260812 02:41:54.532611 6865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:54.533504 6865 log.cc:824] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:54.535392 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:54.531429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:57579 (local address 127.0.0.1:41907)
0812 02:41:54.531869 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:54.531885 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:54.531923 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:41:54.532605 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:54.532623 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:54.532957 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:54.533312 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:54.534137 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:54.534143 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:54.534158 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:54.534172 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:54.534259 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:54.534292 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:54.534417 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:54.534490 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 02:41:54.534650 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":116,"threads_started":1}
I20260812 02:41:54.535768 6865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4: No bootstrap required, opened a new log
I20260812 02:41:54.538378 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:41:54.540236 6865 raft_consensus.cc:348] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec68b27f5ca486e95edbd1c4ae56ac4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41907 } }
I20260812 02:41:54.540522 6865 raft_consensus.cc:374] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:54.540592 6865 raft_consensus.cc:729] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec68b27f5ca486e95edbd1c4ae56ac4, State: Initialized, Role: FOLLOWER
I20260812 02:41:54.541122 6865 consensus_queue.cc:260] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec68b27f5ca486e95edbd1c4ae56ac4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41907 } }
I20260812 02:41:54.541321 6865 raft_consensus.cc:388] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:54.541397 6865 raft_consensus.cc:482] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:54.541515 6865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:54.542346 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:54.540691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:45323 (local address 127.0.0.1:41907)
0812 02:41:54.540808 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:54.540812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:54.541024 (+ 212us) server_negotiation.cc:407] Connection header received
0812 02:41:54.541132 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:54.541134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:54.541191 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:54.541258 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:54.541984 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:54.541986 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:54.541988 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:54.542000 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:54.542028 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:54.542038 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:54.542108 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:54.542199 (+ 91us) server_negotiation.cc:299] Negotiation successful
0812 02:41:54.542231 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 02:41:54.542708 6865 raft_consensus.cc:504] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec68b27f5ca486e95edbd1c4ae56ac4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41907 } }
I20260812 02:41:54.543088 6865 leader_election.cc:302] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dec68b27f5ca486e95edbd1c4ae56ac4; no voters:
I20260812 02:41:54.543438 6865 leader_election.cc:288] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:41:54.543565 6882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:54.543772 6882 raft_consensus.cc:686] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4 [term 1 LEADER]: Becoming Leader. State: Replica: dec68b27f5ca486e95edbd1c4ae56ac4, State: Running, Role: LEADER
I20260812 02:41:54.544071 6882 consensus_queue.cc:237] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec68b27f5ca486e95edbd1c4ae56ac4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41907 } }
I20260812 02:41:54.544683 6865 sys_catalog.cc:564] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:54.545653 6884 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dec68b27f5ca486e95edbd1c4ae56ac4. Latest consensus state: current_term: 1 leader_uuid: "dec68b27f5ca486e95edbd1c4ae56ac4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec68b27f5ca486e95edbd1c4ae56ac4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41907 } } }
I20260812 02:41:54.545784 6884 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:54.546049 6883 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dec68b27f5ca486e95edbd1c4ae56ac4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec68b27f5ca486e95edbd1c4ae56ac4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41907 } } }
I20260812 02:41:54.546139 6883 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:54.546178 6896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:54.548960 6896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:54.555354 6896 catalog_manager.cc:1347] Generated new cluster ID: b5ec3b0c460d4c529275d6495fa2794c
I20260812 02:41:54.555430 6896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:54.570292 6896 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:41:54.570676 6896 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:54.579231 6896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dec68b27f5ca486e95edbd1c4ae56ac4: Generated new TSK 0
I20260812 02:41:54.580108 6896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:41:54.588817 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 6751
2026-08-12T02:41:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1548 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:41:54 dist-test-slave-rmhg krb5kdc[6913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:41:54 dist-test-slave-rmhg krb5kdc[6913](info): set up 2 sockets
Aug 12 02:41:54 dist-test-slave-rmhg krb5kdc[6913](info): commencing operation
krb5kdc: starting...
W20260812 02:41:55.758212 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:41:55 dist-test-slave-rmhg krb5kdc[6913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:41:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:41:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:41:55.890237 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:33893
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:36739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:33893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:56.006147 6935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:56.006454 6935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:56.006516 6935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:56.010651 6935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:56.010737 6935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:56.010766 6935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:56.010789 6935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:56.010819 6935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:56.016500 6935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:36739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:33893
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:33893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:56.017751 6935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:56.018798 6935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:56.028127 6946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:56.028134 6943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:56.028139 6944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:56.028532 6935 server_base.cc:1034] running on GCE node
I20260812 02:41:56.029207 6935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:56.030452 6935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:56.031649 6935 hybrid_clock.cc:648] HybridClock initialized: now 1786502516031628 us; error 37 us; skew 500 ppm
Aug 12 02:41:56 dist-test-slave-rmhg krb5kdc[6913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502516, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:56.034730 6935 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:41:56.035804 6935 webserver.cc:460] Webserver started at http://127.3.127.190:41683/ using document root <none> and password file <none>
I20260812 02:41:56.036394 6935 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:56.036466 6935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:56.036707 6935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:56.039016 6935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "1bc3b272a74b495db259464f2990514a"
format_stamp: "Formatted at 2026-08-12 02:41:56 on dist-test-slave-rmhg"
server_key: "14cdfdda11e6a605f3996313f9a8d33f"
server_key_iv: "fb0d70b444aad21f332ce9be9377e7ad"
server_key_version: "encryptionkey@0"
I20260812 02:41:56.039614 6935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "1bc3b272a74b495db259464f2990514a"
format_stamp: "Formatted at 2026-08-12 02:41:56 on dist-test-slave-rmhg"
server_key: "14cdfdda11e6a605f3996313f9a8d33f"
server_key_iv: "fb0d70b444aad21f332ce9be9377e7ad"
server_key_version: "encryptionkey@0"
I20260812 02:41:56.043742 6935 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260812 02:41:56.046566 6962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:56.047974 6935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:41:56.048130 6935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "1bc3b272a74b495db259464f2990514a"
format_stamp: "Formatted at 2026-08-12 02:41:56 on dist-test-slave-rmhg"
server_key: "14cdfdda11e6a605f3996313f9a8d33f"
server_key_iv: "fb0d70b444aad21f332ce9be9377e7ad"
server_key_version: "encryptionkey@0"
I20260812 02:41:56.048246 6935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:56.076778 6935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:56.080415 6935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:56.080641 6935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:56.088317 6935 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:33893
I20260812 02:41:56.088359 7046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:33893 every 8 connection(s)
I20260812 02:41:56.089407 6935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:56.092489 7047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:56.098506 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 6935
I20260812 02:41:56.098647 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:41:56.098757 7047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a: Bootstrap starting.
I20260812 02:41:56.098912 3582 external_mini_cluster.cc:1428] Setting key 3ee7d7f03bcc8c2fd9b34939d382f915
I20260812 02:41:56.101495 7047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:56.102270 7047 log.cc:824] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:56.104487 7047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a: No bootstrap required, opened a new log
Aug 12 02:41:56 dist-test-slave-rmhg krb5kdc[6913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:41:56.109093 7047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc3b272a74b495db259464f2990514a" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33893 } }
I20260812 02:41:56.109431 7047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:56.109494 7047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bc3b272a74b495db259464f2990514a, State: Initialized, Role: FOLLOWER
I20260812 02:41:56.110040 7047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc3b272a74b495db259464f2990514a" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33893 } }
I20260812 02:41:56.110208 7047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:56.110280 7047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:56.110385 7047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:56.111373 7047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc3b272a74b495db259464f2990514a" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33893 } }
I20260812 02:41:56.111795 7047 leader_election.cc:302] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bc3b272a74b495db259464f2990514a; no voters:
I20260812 02:41:56.112185 7047 leader_election.cc:288] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:41:56.112332 7056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:56.112563 7056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a [term 1 LEADER]: Becoming Leader. State: Replica: 1bc3b272a74b495db259464f2990514a, State: Running, Role: LEADER
I20260812 02:41:56.112913 7056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc3b272a74b495db259464f2990514a" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33893 } }
I20260812 02:41:56.113656 7047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:56.115415 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:56.100454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57533 (local address 127.3.127.190:33893)
0812 02:41:56.100917 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:56.100935 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:56.100975 (+ 40us) server_negotiation.cc:407] Connection header received
0812 02:41:56.101747 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:56.101770 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:56.102148 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:56.102454 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:56.103354 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:56.104423 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:56.105634 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:56.106142 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:56.106267 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:56.109306 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:56.109312 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:56.109327 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:56.109360 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:56.111162 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:56.111743 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:56.111749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:56.111754 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:56.111877 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:56.112253 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:56.112256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:56.112258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:56.112862 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:56.113096 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:56.114019 (+ 923us) server_negotiation.cc:299] Negotiation successful
0812 02:41:56.114269 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":136,"threads_started":1}
I20260812 02:41:56.116297 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bc3b272a74b495db259464f2990514a. Latest consensus state: current_term: 1 leader_uuid: "1bc3b272a74b495db259464f2990514a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc3b272a74b495db259464f2990514a" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33893 } } }
I20260812 02:41:56.117234 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:56.116326 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bc3b272a74b495db259464f2990514a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc3b272a74b495db259464f2990514a" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 33893 } } }
I20260812 02:41:56.117643 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:56.119304 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260812 02:41:56.119500 7079 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:41:56.119573 7079 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:41:56.119642 7081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:56.120565 7081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:56.126744 7081 catalog_manager.cc:1347] Generated new cluster ID: 63c99d00285946f4a2a0a0956d2efde0
I20260812 02:41:56.126814 7081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:56.130553 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:56.122032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47957 (local address 127.3.127.190:33893)
0812 02:41:56.122144 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:56.122148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:56.122163 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:41:56.122202 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:56.122205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:56.122250 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:56.122337 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:56.123051 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:56.123588 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:56.124388 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:56.124668 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:56.124772 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:56.126519 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:56.126523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:56.126526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:41:56.126552 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:56.128207 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:56.128823 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:56.128827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:56.128829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:56.128919 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:41:56.129401 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:41:56.129404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:41:56.129406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:41:56.129635 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:56.129758 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:56.130278 (+ 520us) server_negotiation.cc:299] Negotiation successful
0812 02:41:56.130387 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 02:41:56.142557 7081 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:41:56.143599 7081 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:56.149355 7081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bc3b272a74b495db259464f2990514a: Generated new TSK 0
I20260812 02:41:56.150247 7081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:41:56.158442 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 6935
W20260812 02:41:56.178987 6911 connection.cc:535] client connection to 127.3.127.190:33893 recv error: Network error: failed to read from TLS socket (remote: 127.3.127.190:33893): Connection reset by peer (error 104)
2026-08-12T02:41:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1576 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:41:56 dist-test-slave-rmhg krb5kdc[7099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:41:56 dist-test-slave-rmhg krb5kdc[7099](info): set up 2 sockets
Aug 12 02:41:56 dist-test-slave-rmhg krb5kdc[7099](info): commencing operation
krb5kdc: starting...
W20260812 02:41:57.345376 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:41:57 dist-test-slave-rmhg krb5kdc[7099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:41:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:41:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:41:57.476027 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:36869
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:40779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:36869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:57.594884 7119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:57.595204 7119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:57.595252 7119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:57.599488 7119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:57.599570 7119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:57.599589 7119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:57.599611 7119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:57.599625 7119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:57.605381 7119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:40779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:36869
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:36869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:57.606618 7119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:57.607712 7119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:57.616813 7125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:57.616804 7127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:57.616861 7119 server_base.cc:1034] running on GCE node
W20260812 02:41:57.616804 7131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:57.617731 7119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:57.618881 7119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:57.620059 7119 hybrid_clock.cc:648] HybridClock initialized: now 1786502517620039 us; error 37 us; skew 500 ppm
Aug 12 02:41:57 dist-test-slave-rmhg krb5kdc[7099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502517, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:57.622816 7119 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:41:57.623754 7119 webserver.cc:460] Webserver started at http://127.3.127.190:36717/ using document root <none> and password file <none>
I20260812 02:41:57.624311 7119 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:57.624370 7119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:57.624591 7119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:57.626722 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "90bd72952945411ea95111f0e3875325"
format_stamp: "Formatted at 2026-08-12 02:41:57 on dist-test-slave-rmhg"
server_key: "eebd2a841d6c85acb3ca47dc93725a5e"
server_key_iv: "c8b4dd0224c3546e2e3f35ed641dd180"
server_key_version: "encryptionkey@0"
I20260812 02:41:57.627293 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "90bd72952945411ea95111f0e3875325"
format_stamp: "Formatted at 2026-08-12 02:41:57 on dist-test-slave-rmhg"
server_key: "eebd2a841d6c85acb3ca47dc93725a5e"
server_key_iv: "c8b4dd0224c3546e2e3f35ed641dd180"
server_key_version: "encryptionkey@0"
I20260812 02:41:57.631263 7119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:41:57.634050 7142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:57.635372 7119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:41:57.635502 7119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "90bd72952945411ea95111f0e3875325"
format_stamp: "Formatted at 2026-08-12 02:41:57 on dist-test-slave-rmhg"
server_key: "eebd2a841d6c85acb3ca47dc93725a5e"
server_key_iv: "c8b4dd0224c3546e2e3f35ed641dd180"
server_key_version: "encryptionkey@0"
I20260812 02:41:57.635620 7119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:57.655609 7119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:57.658957 7119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:57.659413 7119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:57.667402 7119 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:36869
I20260812 02:41:57.667421 7224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:36869 every 8 connection(s)
I20260812 02:41:57.668460 7119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:57.671614 7226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:57.674372 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 7119
I20260812 02:41:57.674535 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:41:57.674845 3582 external_mini_cluster.cc:1428] Setting key c49700ae3746af8699e06df6b9587074
I20260812 02:41:57.678534 7226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325: Bootstrap starting.
I20260812 02:41:57.681339 7226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:57.682258 7226 log.cc:824] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:57.684651 7226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325: No bootstrap required, opened a new log
I20260812 02:41:57.684803 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:57.676419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49523 (local address 127.3.127.190:36869)
0812 02:41:57.676881 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:57.676898 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:57.676940 (+ 42us) server_negotiation.cc:407] Connection header received
0812 02:41:57.677685 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:57.677707 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:57.678115 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:57.678437 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:57.678842 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:57.679867 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:57.681119 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:57.681735 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:57.681877 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:57.682840 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:57.682847 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:57.682865 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:57.683363 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:57.683422 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:57.683447 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:57.683559 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:57.683807 (+ 248us) server_negotiation.cc:299] Negotiation successful
0812 02:41:57.683965 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":135,"threads_started":1}
I20260812 02:41:57.687961 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:41:57.688728 7226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bd72952945411ea95111f0e3875325" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36869 } }
I20260812 02:41:57.688972 7226 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:57.689041 7226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90bd72952945411ea95111f0e3875325, State: Initialized, Role: FOLLOWER
I20260812 02:41:57.689530 7226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bd72952945411ea95111f0e3875325" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36869 } }
I20260812 02:41:57.689683 7226 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:57.689801 7226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:57.689924 7226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:57.691002 7226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bd72952945411ea95111f0e3875325" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36869 } }
I20260812 02:41:57.691350 7226 leader_election.cc:302] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90bd72952945411ea95111f0e3875325; no voters:
I20260812 02:41:57.691668 7226 leader_election.cc:288] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:41:57.691761 7245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:57.691931 7245 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325 [term 1 LEADER]: Becoming Leader. State: Replica: 90bd72952945411ea95111f0e3875325, State: Running, Role: LEADER
I20260812 02:41:57.692327 7245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bd72952945411ea95111f0e3875325" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36869 } }
I20260812 02:41:57.693109 7226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:57.693645 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90bd72952945411ea95111f0e3875325. Latest consensus state: current_term: 1 leader_uuid: "90bd72952945411ea95111f0e3875325" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bd72952945411ea95111f0e3875325" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36869 } } }
I20260812 02:41:57.693800 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:57.694460 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:57.690045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42237 (local address 127.3.127.190:36869)
0812 02:41:57.690163 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:57.690167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:57.690392 (+ 225us) server_negotiation.cc:407] Connection header received
0812 02:41:57.690524 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:57.690527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:57.690574 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:57.690643 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:57.690922 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:57.691574 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:57.692631 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:41:57.692893 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:41:57.693002 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:41:57.693881 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:57.693885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:57.693888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:57.693903 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:57.693934 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:57.693947 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:57.694025 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:57.694283 (+ 258us) server_negotiation.cc:299] Negotiation successful
0812 02:41:57.694323 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 02:41:57.694901 7246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90bd72952945411ea95111f0e3875325" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bd72952945411ea95111f0e3875325" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 36869 } } }
I20260812 02:41:57.695006 7246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:57.695564 7255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:57.698751 7255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:57.705041 7255 catalog_manager.cc:1347] Generated new cluster ID: 574839f8f90c435895d7253376866b2d
I20260812 02:41:57.705140 7255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:57.716005 7255 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:41:57.717039 7255 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:57.723765 7255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90bd72952945411ea95111f0e3875325: Generated new TSK 0
I20260812 02:41:57.724915 7255 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:41:57.736768 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 7119
W20260812 02:41:57.752660 7094 connection.cc:535] client connection to 127.3.127.190:36869 recv error: Network error: failed to read from TLS socket (remote: 127.3.127.190:36869): Connection reset by peer (error 104)
2026-08-12T02:41:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1573 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:41:57 dist-test-slave-rmhg krb5kdc[7283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:41:57 dist-test-slave-rmhg krb5kdc[7283](info): set up 2 sockets
Aug 12 02:41:57 dist-test-slave-rmhg krb5kdc[7283](info): commencing operation
krb5kdc: starting...
W20260812 02:41:58.904538 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:41:58 dist-test-slave-rmhg krb5kdc[7283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:41:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:41:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:41:59.036382 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:35417
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:39019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:35417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:41:59.151229 7299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:41:59.151543 7299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:41:59.151585 7299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:41:59.155685 7299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:41:59.155755 7299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:41:59.155773 7299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:41:59.155789 7299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:41:59.155812 7299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:41:59.161216 7299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:35417
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:35417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:41:59.162357 7299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:41:59.163486 7299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:41:59.172271 7307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:59.172444 7299 server_base.cc:1034] running on GCE node
W20260812 02:41:59.172398 7311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:41:59.172398 7308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:41:59.173308 7299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:41:59.174516 7299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:41:59.175709 7299 hybrid_clock.cc:648] HybridClock initialized: now 1786502519175692 us; error 34 us; skew 500 ppm
Aug 12 02:41:59 dist-test-slave-rmhg krb5kdc[7283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502519, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:41:59.178695 7299 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:41:59.179651 7299 webserver.cc:460] Webserver started at http://127.3.127.190:39147/ using document root <none> and password file <none>
I20260812 02:41:59.180223 7299 fs_manager.cc:359] Metadata directory not provided
I20260812 02:41:59.180302 7299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:41:59.180523 7299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:41:59.182670 7299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "d939e2eed0d1486db847987abcc82957"
format_stamp: "Formatted at 2026-08-12 02:41:59 on dist-test-slave-rmhg"
server_key: "a2efcbc1b6b57bf2751e27d2309391b9"
server_key_iv: "0981cb85c8dee16c0cabb21c2f2ce974"
server_key_version: "encryptionkey@0"
I20260812 02:41:59.183274 7299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "d939e2eed0d1486db847987abcc82957"
format_stamp: "Formatted at 2026-08-12 02:41:59 on dist-test-slave-rmhg"
server_key: "a2efcbc1b6b57bf2751e27d2309391b9"
server_key_iv: "0981cb85c8dee16c0cabb21c2f2ce974"
server_key_version: "encryptionkey@0"
I20260812 02:41:59.187268 7299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:41:59.189877 7325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:41:59.191234 7299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:41:59.191360 7299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "d939e2eed0d1486db847987abcc82957"
format_stamp: "Formatted at 2026-08-12 02:41:59 on dist-test-slave-rmhg"
server_key: "a2efcbc1b6b57bf2751e27d2309391b9"
server_key_iv: "0981cb85c8dee16c0cabb21c2f2ce974"
server_key_version: "encryptionkey@0"
I20260812 02:41:59.191466 7299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:41:59.196034 7299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:41:59.199189 7299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:41:59.199359 7299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:41:59.206228 7299 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:35417
I20260812 02:41:59.206269 7416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:35417 every 8 connection(s)
I20260812 02:41:59.207202 7299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:41:59.210191 7417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:41:59.213575 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 7299
I20260812 02:41:59.213713 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:41:59.213981 3582 external_mini_cluster.cc:1428] Setting key 88c5e1eb9c9f51d85f340df81ab9bb93
I20260812 02:41:59.216619 7417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957: Bootstrap starting.
I20260812 02:41:59.219620 7417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957: Neither blocks nor log segments found. Creating new log.
I20260812 02:41:59.219937 7422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:59.215479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42255 (local address 127.3.127.190:35417)
0812 02:41:59.215975 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:59.215997 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:59.216046 (+ 49us) server_negotiation.cc:407] Connection header received
0812 02:41:59.216917 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:59.216942 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:59.217376 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:59.217766 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:59.218446 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:59.218452 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:59.218467 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:59.218477 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:59.218534 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:59.218559 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:59.218699 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:59.218767 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 02:41:59.218945 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":143,"threads_started":1}
I20260812 02:41:59.220506 7417 log.cc:824] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957: Log is configured to *not* fsync() on all Append() calls
I20260812 02:41:59.222707 7417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957: No bootstrap required, opened a new log
I20260812 02:41:59.222884 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:41:59.226500 7422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:41:59.224871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55531 (local address 127.3.127.190:35417)
0812 02:41:59.225007 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 02:41:59.225011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:41:59.225234 (+ 223us) server_negotiation.cc:407] Connection header received
0812 02:41:59.225365 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:41:59.225368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:41:59.225402 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:41:59.225463 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:41:59.226133 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:41:59.226136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:41:59.226139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:41:59.226151 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:41:59.226180 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:41:59.226190 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:41:59.226260 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 02:41:59.226337 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 02:41:59.226378 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 02:41:59.226666 7417 raft_consensus.cc:348] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d939e2eed0d1486db847987abcc82957" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 35417 } }
I20260812 02:41:59.226967 7417 raft_consensus.cc:374] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:41:59.227023 7417 raft_consensus.cc:729] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d939e2eed0d1486db847987abcc82957, State: Initialized, Role: FOLLOWER
I20260812 02:41:59.227484 7417 consensus_queue.cc:260] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d939e2eed0d1486db847987abcc82957" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 35417 } }
I20260812 02:41:59.227628 7417 raft_consensus.cc:388] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:41:59.227695 7417 raft_consensus.cc:482] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:41:59.227798 7417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:41:59.228847 7417 raft_consensus.cc:504] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d939e2eed0d1486db847987abcc82957" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 35417 } }
I20260812 02:41:59.229235 7417 leader_election.cc:302] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d939e2eed0d1486db847987abcc82957; no voters:
I20260812 02:41:59.229552 7417 leader_election.cc:288] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:41:59.229651 7435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:41:59.229861 7435 raft_consensus.cc:686] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957 [term 1 LEADER]: Becoming Leader. State: Replica: d939e2eed0d1486db847987abcc82957, State: Running, Role: LEADER
I20260812 02:41:59.230227 7435 consensus_queue.cc:237] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d939e2eed0d1486db847987abcc82957" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 35417 } }
I20260812 02:41:59.230813 7417 sys_catalog.cc:564] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:41:59.231950 7436 sys_catalog.cc:455] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d939e2eed0d1486db847987abcc82957" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d939e2eed0d1486db847987abcc82957" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 35417 } } }
I20260812 02:41:59.231950 7439 sys_catalog.cc:455] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d939e2eed0d1486db847987abcc82957. Latest consensus state: current_term: 1 leader_uuid: "d939e2eed0d1486db847987abcc82957" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d939e2eed0d1486db847987abcc82957" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 35417 } } }
I20260812 02:41:59.232131 7436 sys_catalog.cc:458] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:59.232156 7439 sys_catalog.cc:458] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957 [sys.catalog]: This master's current role is: LEADER
I20260812 02:41:59.232645 7452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:41:59.235658 7452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:41:59.241642 7452 catalog_manager.cc:1347] Generated new cluster ID: c403e4728aa141379ab4d7f17d011dea
I20260812 02:41:59.241726 7452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:41:59.254619 7452 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:41:59.254961 7452 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:41:59.272140 7452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d939e2eed0d1486db847987abcc82957: Generated new TSK 0
I20260812 02:41:59.273118 7452 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:41:59.282508 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 7299
2026-08-12T02:41:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1545 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:41:59 dist-test-slave-rmhg krb5kdc[7469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:41:59 dist-test-slave-rmhg krb5kdc[7469](info): set up 2 sockets
Aug 12 02:41:59 dist-test-slave-rmhg krb5kdc[7469](info): commencing operation
krb5kdc: starting...
W20260812 02:42:00.464327 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:42:00 dist-test-slave-rmhg krb5kdc[7469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:42:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:42:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:42:00.598619 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:35085
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:35085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:42:00.718950 7489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:42:00.719250 7489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:42:00.719314 7489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:42:00.723438 7489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:42:00.723549 7489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:42:00.723583 7489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:42:00.723606 7489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:42:00.723631 7489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:42:00.729368 7489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:35085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:35085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:42:00.730547 7489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:42:00.731611 7489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:42:00.740473 7501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:42:00.740509 7500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:42:00.740530 7489 server_base.cc:1034] running on GCE node
W20260812 02:42:00.740474 7505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:42:00.741317 7489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:42:00.742589 7489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:42:00.743783 7489 hybrid_clock.cc:648] HybridClock initialized: now 1786502520743764 us; error 34 us; skew 500 ppm
Aug 12 02:42:00 dist-test-slave-rmhg krb5kdc[7469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502520, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:42:00.751323 7489 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:42:00.752458 7489 webserver.cc:460] Webserver started at http://127.3.127.190:37357/ using document root <none> and password file <none>
I20260812 02:42:00.753086 7489 fs_manager.cc:359] Metadata directory not provided
I20260812 02:42:00.753188 7489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:42:00.753424 7489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:42:00.755671 7489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "aded358df6294fcb83c61fd61685256a"
format_stamp: "Formatted at 2026-08-12 02:42:00 on dist-test-slave-rmhg"
server_key: "cd6cfb8c9f6a9c130ac75f76ac4f8d2a"
server_key_iv: "6e08200ad0322582dccb8e652e507053"
server_key_version: "encryptionkey@0"
I20260812 02:42:00.756328 7489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "aded358df6294fcb83c61fd61685256a"
format_stamp: "Formatted at 2026-08-12 02:42:00 on dist-test-slave-rmhg"
server_key: "cd6cfb8c9f6a9c130ac75f76ac4f8d2a"
server_key_iv: "6e08200ad0322582dccb8e652e507053"
server_key_version: "encryptionkey@0"
I20260812 02:42:00.760483 7489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:42:00.763401 7518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:42:00.764879 7489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:42:00.765038 7489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "aded358df6294fcb83c61fd61685256a"
format_stamp: "Formatted at 2026-08-12 02:42:00 on dist-test-slave-rmhg"
server_key: "cd6cfb8c9f6a9c130ac75f76ac4f8d2a"
server_key_iv: "6e08200ad0322582dccb8e652e507053"
server_key_version: "encryptionkey@0"
I20260812 02:42:00.765190 7489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:42:00.779537 7489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:42:00.783245 7489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:42:00.783460 7489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:42:00.790922 7489 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:35085
I20260812 02:42:00.790940 7619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:35085 every 8 connection(s)
I20260812 02:42:00.791983 7489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:42:00.795014 7620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:42:00.797026 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 7489
I20260812 02:42:00.797204 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:42:00.797488 3582 external_mini_cluster.cc:1428] Setting key e746d1a6b540b63920ed755c8665a700
I20260812 02:42:00.802193 7620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a: Bootstrap starting.
I20260812 02:42:00.804979 7620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a: Neither blocks nor log segments found. Creating new log.
I20260812 02:42:00.805980 7620 log.cc:824] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:42:00 dist-test-slave-rmhg krb5kdc[7469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:42:00.808362 7620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a: No bootstrap required, opened a new log
I20260812 02:42:00.812218 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:00.798981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34661 (local address 127.3.127.190:35085)
0812 02:42:00.799436 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:00.799453 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:00.799491 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:42:00.800165 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:00.800186 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:00.800561 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:00.800872 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:00.801883 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:00.802947 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:00.804143 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:00.804644 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:00.804792 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:00.807565 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:00.807572 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:00.807589 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:00.807622 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:00.809362 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:00.809833 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:00.809837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:00.809842 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:00.809955 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:00.810233 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:00.810235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:00.810236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:00.810641 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:00.810789 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:00.811087 (+ 298us) server_negotiation.cc:299] Negotiation successful
0812 02:42:00.811388 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":140,"threads_started":1}
I20260812 02:42:00.812494 7620 raft_consensus.cc:348] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aded358df6294fcb83c61fd61685256a" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 35085 } }
I20260812 02:42:00.812748 7620 raft_consensus.cc:374] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:42:00.812803 7620 raft_consensus.cc:729] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aded358df6294fcb83c61fd61685256a, State: Initialized, Role: FOLLOWER
I20260812 02:42:00.813292 7620 consensus_queue.cc:260] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aded358df6294fcb83c61fd61685256a" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 35085 } }
I20260812 02:42:00.813432 7620 raft_consensus.cc:388] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:42:00.813498 7620 raft_consensus.cc:482] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:42:00.813597 7620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:42:00.814678 7620 raft_consensus.cc:504] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aded358df6294fcb83c61fd61685256a" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 35085 } }
I20260812 02:42:00.815013 7620 leader_election.cc:302] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aded358df6294fcb83c61fd61685256a; no voters:
I20260812 02:42:00.815219 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:42:00.815400 7620 leader_election.cc:288] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:42:00.815491 7627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:42:00.815708 7627 raft_consensus.cc:686] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a [term 1 LEADER]: Becoming Leader. State: Replica: aded358df6294fcb83c61fd61685256a, State: Running, Role: LEADER
I20260812 02:42:00.816107 7627 consensus_queue.cc:237] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aded358df6294fcb83c61fd61685256a" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 35085 } }
I20260812 02:42:00.816697 7620 sys_catalog.cc:564] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:42:00.817432 7629 sys_catalog.cc:455] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a [sys.catalog]: SysCatalogTable state changed. Reason: New leader aded358df6294fcb83c61fd61685256a. Latest consensus state: current_term: 1 leader_uuid: "aded358df6294fcb83c61fd61685256a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aded358df6294fcb83c61fd61685256a" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 35085 } } }
I20260812 02:42:00.817616 7629 sys_catalog.cc:458] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a [sys.catalog]: This master's current role is: LEADER
I20260812 02:42:00.817914 7628 sys_catalog.cc:455] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aded358df6294fcb83c61fd61685256a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aded358df6294fcb83c61fd61685256a" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 35085 } } }
I20260812 02:42:00.818017 7628 sys_catalog.cc:458] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a [sys.catalog]: This master's current role is: LEADER
I20260812 02:42:00.818089 7645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:42:00.821591 7645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:42:00.827930 7645 catalog_manager.cc:1347] Generated new cluster ID: 7be69eca56d74eb3a447b4df49b5ca17
I20260812 02:42:00.828061 7645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:42:00.828392 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:00.818263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58587 (local address 127.3.127.190:35085)
0812 02:42:00.818377 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:00.818382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:00.818920 (+ 538us) server_negotiation.cc:407] Connection header received
0812 02:42:00.819057 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:00.819060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:00.819124 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:00.819203 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:00.820089 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:00.820839 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:00.822084 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:00.822426 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:00.822510 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:00.824611 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:00.824614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:00.824617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:00.824648 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:00.826273 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:00.826683 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:00.826687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:00.826689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:00.826779 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:00.827049 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:00.827052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:00.827054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:00.827262 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:00.827377 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:00.828113 (+ 736us) server_negotiation.cc:299] Negotiation successful
0812 02:42:00.828245 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 02:42:00.851960 7645 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:42:00.853246 7645 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:42:00.862430 7645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aded358df6294fcb83c61fd61685256a: Generated new TSK 0
I20260812 02:42:00.863373 7645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:42:00.872053 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 7489
2026-08-12T02:42:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1588 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:42:00 dist-test-slave-rmhg krb5kdc[7664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:42:00 dist-test-slave-rmhg krb5kdc[7664](info): set up 2 sockets
Aug 12 02:42:00 dist-test-slave-rmhg krb5kdc[7664](info): commencing operation
krb5kdc: starting...
W20260812 02:42:02.049336 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:42:02 dist-test-slave-rmhg krb5kdc[7664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:42:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:42:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:42:02.190335 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:38977
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:43429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:38977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:42:02.306078 7684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:42:02.306399 7684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:42:02.306465 7684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:42:02.310688 7684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:42:02.310777 7684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:42:02.310803 7684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:42:02.310827 7684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:42:02.310858 7684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:42:02.316493 7684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:43429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:38977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:38977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:42:02.317734 7684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:42:02.318801 7684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:42:02.327538 7697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:42:02.327538 7700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:42:02.327555 7696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:42:02.327879 7684 server_base.cc:1034] running on GCE node
I20260812 02:42:02.328500 7684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:42:02.329833 7684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:42:02.331022 7684 hybrid_clock.cc:648] HybridClock initialized: now 1786502522330999 us; error 37 us; skew 500 ppm
Aug 12 02:42:02 dist-test-slave-rmhg krb5kdc[7664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502522, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:42:02.333933 7684 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:42:02.334957 7684 webserver.cc:460] Webserver started at http://127.3.127.190:43179/ using document root <none> and password file <none>
I20260812 02:42:02.335541 7684 fs_manager.cc:359] Metadata directory not provided
I20260812 02:42:02.335613 7684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:42:02.335829 7684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:42:02.338050 7684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "c3c5aa11ebbd4f4abc4cf33f9640b56c"
format_stamp: "Formatted at 2026-08-12 02:42:02 on dist-test-slave-rmhg"
server_key: "bdf7bc834d2edb3e79160711cbaef5f0"
server_key_iv: "2a127db770307f453d279ef0b9cfb0a8"
server_key_version: "encryptionkey@0"
I20260812 02:42:02.338618 7684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "c3c5aa11ebbd4f4abc4cf33f9640b56c"
format_stamp: "Formatted at 2026-08-12 02:42:02 on dist-test-slave-rmhg"
server_key: "bdf7bc834d2edb3e79160711cbaef5f0"
server_key_iv: "2a127db770307f453d279ef0b9cfb0a8"
server_key_version: "encryptionkey@0"
I20260812 02:42:02.342644 7684 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:42:02.345383 7714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:42:02.346828 7684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:42:02.346978 7684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "c3c5aa11ebbd4f4abc4cf33f9640b56c"
format_stamp: "Formatted at 2026-08-12 02:42:02 on dist-test-slave-rmhg"
server_key: "bdf7bc834d2edb3e79160711cbaef5f0"
server_key_iv: "2a127db770307f453d279ef0b9cfb0a8"
server_key_version: "encryptionkey@0"
I20260812 02:42:02.347126 7684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:42:02.367502 7684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:42:02.370757 7684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:42:02.370970 7684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:42:02.378801 7684 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:38977
I20260812 02:42:02.378931 7817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:38977 every 8 connection(s)
I20260812 02:42:02.380057 7684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:42:02.383649 7818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:42:02.388306 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 7684
I20260812 02:42:02.388463 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:42:02.388778 3582 external_mini_cluster.cc:1428] Setting key 97dd96a96704f114533c2d3be184dfda
I20260812 02:42:02.390475 7818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c: Bootstrap starting.
I20260812 02:42:02.392863 7818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c: Neither blocks nor log segments found. Creating new log.
I20260812 02:42:02.393700 7818 log.cc:824] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c: Log is configured to *not* fsync() on all Append() calls
I20260812 02:42:02.396042 7818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c: No bootstrap required, opened a new log
I20260812 02:42:02.398684 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:02.390348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43325 (local address 127.3.127.190:38977)
0812 02:42:02.390884 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:02.390907 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:02.390955 (+ 48us) server_negotiation.cc:407] Connection header received
0812 02:42:02.391759 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:02.391778 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:02.392145 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:02.392488 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:02.392963 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:02.394108 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:02.395353 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:02.395887 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:02.396008 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:02.396891 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:02.396897 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:02.396911 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:42:02.397374 (+ 463us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:02.397433 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:42:02.397457 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:02.397535 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:02.397700 (+ 165us) server_negotiation.cc:299] Negotiation successful
0812 02:42:02.397913 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":152,"threads_started":1}
I20260812 02:42:02.400054 7818 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c5aa11ebbd4f4abc4cf33f9640b56c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38977 } }
I20260812 02:42:02.400306 7818 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:42:02.400345 7818 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3c5aa11ebbd4f4abc4cf33f9640b56c, State: Initialized, Role: FOLLOWER
I20260812 02:42:02.400810 7818 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c5aa11ebbd4f4abc4cf33f9640b56c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38977 } }
I20260812 02:42:02.400985 7818 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:42:02.401058 7818 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:42:02.401185 7818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:42:02.401618 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:42:02.402269 7818 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c5aa11ebbd4f4abc4cf33f9640b56c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38977 } }
I20260812 02:42:02.402611 7818 leader_election.cc:302] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3c5aa11ebbd4f4abc4cf33f9640b56c; no voters:
I20260812 02:42:02.402961 7818 leader_election.cc:288] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:42:02.403060 7829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:42:02.403247 7829 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c [term 1 LEADER]: Becoming Leader. State: Replica: c3c5aa11ebbd4f4abc4cf33f9640b56c, State: Running, Role: LEADER
I20260812 02:42:02.404104 7829 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c5aa11ebbd4f4abc4cf33f9640b56c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38977 } }
I20260812 02:42:02.404364 7818 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:42:02.405825 7832 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3c5aa11ebbd4f4abc4cf33f9640b56c. Latest consensus state: current_term: 1 leader_uuid: "c3c5aa11ebbd4f4abc4cf33f9640b56c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c5aa11ebbd4f4abc4cf33f9640b56c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38977 } } }
I20260812 02:42:02.405962 7832 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c [sys.catalog]: This master's current role is: LEADER
I20260812 02:42:02.406307 7831 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3c5aa11ebbd4f4abc4cf33f9640b56c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c5aa11ebbd4f4abc4cf33f9640b56c" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 38977 } } }
I20260812 02:42:02.406392 7831 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c [sys.catalog]: This master's current role is: LEADER
I20260812 02:42:02.408264 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:02.403720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55825 (local address 127.3.127.190:38977)
0812 02:42:02.403816 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:02.403821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:02.404022 (+ 201us) server_negotiation.cc:407] Connection header received
0812 02:42:02.404131 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:02.404134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:02.404179 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:02.404246 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:02.404633 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:02.405216 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:02.406302 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:02.406615 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:02.406720 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:02.407811 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:02.407814 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:02.407817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:42:02.407830 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:02.407857 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:42:02.407868 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:02.407930 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:02.408079 (+ 149us) server_negotiation.cc:299] Negotiation successful
0812 02:42:02.408139 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260812 02:42:02.409557 7851 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:42:02.409624 7851 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:42:02.409696 7847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:42:02.410410 7847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:42:02.416280 7847 catalog_manager.cc:1347] Generated new cluster ID: da823f46acfa4f17bb4513c95a54606f
I20260812 02:42:02.416340 7847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:42:02.431236 7847 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:42:02.432251 7847 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:42:02.437417 7847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3c5aa11ebbd4f4abc4cf33f9640b56c: Generated new TSK 0
I20260812 02:42:02.438144 7847 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:42:02.450896 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 7684
2026-08-12T02:42:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1578 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:42:02 dist-test-slave-rmhg krb5kdc[7861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:42:02 dist-test-slave-rmhg krb5kdc[7861](info): set up 2 sockets
Aug 12 02:42:02 dist-test-slave-rmhg krb5kdc[7861](info): commencing operation
krb5kdc: starting...
W20260812 02:42:03.629944 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:42:03 dist-test-slave-rmhg krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:42:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:42:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:42:03.760507 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:45255
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:44999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:45255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:42:03.878923 7878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:42:03.879222 7878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:42:03.879266 7878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:42:03.883312 7878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:42:03.883390 7878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:42:03.883409 7878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:42:03.883425 7878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:42:03.883440 7878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:42:03.888922 7878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:44999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:45255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:45255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:42:03.890167 7878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:42:03.891170 7878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:42:03.899685 7890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:42:03.899706 7894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:42:03.899696 7887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:42:03.899813 7878 server_base.cc:1034] running on GCE node
I20260812 02:42:03.900681 7878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:42:03.901921 7878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:42:03.903107 7878 hybrid_clock.cc:648] HybridClock initialized: now 1786502523903089 us; error 28 us; skew 500 ppm
Aug 12 02:42:03 dist-test-slave-rmhg krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502523, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:42:03.905910 7878 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:42:03.906904 7878 webserver.cc:460] Webserver started at http://127.3.127.190:35677/ using document root <none> and password file <none>
I20260812 02:42:03.907465 7878 fs_manager.cc:359] Metadata directory not provided
I20260812 02:42:03.907538 7878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:42:03.907743 7878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:42:03.909899 7878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "1456250aa631435b98308cecb9934c66"
format_stamp: "Formatted at 2026-08-12 02:42:03 on dist-test-slave-rmhg"
server_key: "09ab3113ad33decde6d264b37adbc0eb"
server_key_iv: "edb0631296b92fcbcf53f6f9c19a8733"
server_key_version: "encryptionkey@0"
I20260812 02:42:03.910465 7878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "1456250aa631435b98308cecb9934c66"
format_stamp: "Formatted at 2026-08-12 02:42:03 on dist-test-slave-rmhg"
server_key: "09ab3113ad33decde6d264b37adbc0eb"
server_key_iv: "edb0631296b92fcbcf53f6f9c19a8733"
server_key_version: "encryptionkey@0"
I20260812 02:42:03.914485 7878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260812 02:42:03.917364 7903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:42:03.918694 7878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:42:03.918828 7878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "1456250aa631435b98308cecb9934c66"
format_stamp: "Formatted at 2026-08-12 02:42:03 on dist-test-slave-rmhg"
server_key: "09ab3113ad33decde6d264b37adbc0eb"
server_key_iv: "edb0631296b92fcbcf53f6f9c19a8733"
server_key_version: "encryptionkey@0"
I20260812 02:42:03.918937 7878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:42:03.923851 7878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:42:03.926937 7878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:42:03.927131 7878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:42:03.934310 7878 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:45255
I20260812 02:42:03.934333 7992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:45255 every 8 connection(s)
I20260812 02:42:03.935264 7878 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:42:03.938272 7993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:42:03.938254 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 7878
I20260812 02:42:03.938410 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:42:03.938774 3582 external_mini_cluster.cc:1428] Setting key 23811b398719f4e7ccf84e9950f1eac1
I20260812 02:42:03.944509 7997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:03.940140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58741 (local address 127.3.127.190:45255)
0812 02:42:03.940568 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:03.940584 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:03.940622 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:42:03.941601 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:03.941622 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:03.941932 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:03.942248 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:03.943180 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:03.943185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:03.943198 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:42:03.943211 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:03.943274 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:42:03.943296 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:03.943430 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:03.943533 (+ 103us) server_negotiation.cc:299] Negotiation successful
0812 02:42:03.943688 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":126,"threads_started":1}
I20260812 02:42:03.945488 7993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66: Bootstrap starting.
I20260812 02:42:03.947692 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:42:03.948189 7993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66: Neither blocks nor log segments found. Creating new log.
I20260812 02:42:03.949126 7993 log.cc:824] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66: Log is configured to *not* fsync() on all Append() calls
I20260812 02:42:03.951697 7993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66: No bootstrap required, opened a new log
I20260812 02:42:03.951833 7997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:03.950120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56231 (local address 127.3.127.190:45255)
0812 02:42:03.950250 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:03.950255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:03.950435 (+ 180us) server_negotiation.cc:407] Connection header received
0812 02:42:03.950577 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:03.950579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:03.950610 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:03.950711 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:03.951401 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:03.951404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:03.951406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:42:03.951415 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:03.951439 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:42:03.951449 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:03.951515 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:03.951614 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 02:42:03.951645 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 02:42:03.955590 7993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1456250aa631435b98308cecb9934c66" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 45255 } }
I20260812 02:42:03.955858 7993 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:42:03.955920 7993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1456250aa631435b98308cecb9934c66, State: Initialized, Role: FOLLOWER
I20260812 02:42:03.956380 7993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1456250aa631435b98308cecb9934c66" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 45255 } }
I20260812 02:42:03.956537 7993 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:42:03.956610 7993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:42:03.956724 7993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:42:03.957784 7993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1456250aa631435b98308cecb9934c66" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 45255 } }
I20260812 02:42:03.958153 7993 leader_election.cc:302] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1456250aa631435b98308cecb9934c66; no voters:
I20260812 02:42:03.958510 7993 leader_election.cc:288] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:42:03.958671 8013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:42:03.958890 8013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66 [term 1 LEADER]: Becoming Leader. State: Replica: 1456250aa631435b98308cecb9934c66, State: Running, Role: LEADER
I20260812 02:42:03.959228 8013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1456250aa631435b98308cecb9934c66" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 45255 } }
I20260812 02:42:03.959815 7993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:42:03.960867 8017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1456250aa631435b98308cecb9934c66. Latest consensus state: current_term: 1 leader_uuid: "1456250aa631435b98308cecb9934c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1456250aa631435b98308cecb9934c66" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 45255 } } }
I20260812 02:42:03.961011 8017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66 [sys.catalog]: This master's current role is: LEADER
I20260812 02:42:03.961292 8016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1456250aa631435b98308cecb9934c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1456250aa631435b98308cecb9934c66" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 45255 } } }
I20260812 02:42:03.961398 8016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66 [sys.catalog]: This master's current role is: LEADER
I20260812 02:42:03.961735 8030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:42:03.965008 8030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:42:03.970957 8030 catalog_manager.cc:1347] Generated new cluster ID: d63271fe754a46a28ebb94e874666fd1
I20260812 02:42:03.971047 8030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:42:03.986531 8030 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:42:03.986950 8030 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:42:03.996637 8030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1456250aa631435b98308cecb9934c66: Generated new TSK 0
I20260812 02:42:03.997586 8030 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:42:04.008782 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 7878
2026-08-12T02:42:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1553 ms)
[----------] 18 tests from AuthTokenIssuingTest (28314 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:42:04 dist-test-slave-rmhg krb5kdc[8048](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:42:04 dist-test-slave-rmhg krb5kdc[8048](info): set up 2 sockets
Aug 12 02:42:04 dist-test-slave-rmhg krb5kdc[8048](info): commencing operation
krb5kdc: starting...
W20260812 02:42:05.175340 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:42:05 dist-test-slave-rmhg krb5kdc[8048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:42:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:42:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:42:05.308349 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:40073
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:40073,127.3.127.189:46427,127.3.127.188:43211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:42:05.426822 8067 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 02:42:05.427147 8067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:42:05.427186 8067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:42:05.427250 8067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:42:05.431347 8067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:42:05.431423 8067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:42:05.431450 8067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:42:05.431470 8067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:42:05.431492 8067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:42:05.437129 8067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:40073,127.3.127.189:46427,127.3.127.188:43211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:40073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:42:05.438305 8067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:42:05.439363 8067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:42:05.448120 8080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:42:05.448141 8078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:42:05.448436 8067 server_base.cc:1034] running on GCE node
W20260812 02:42:05.448120 8073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:42:05.449242 8067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:42:05.450489 8067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:42:05.451691 8067 hybrid_clock.cc:648] HybridClock initialized: now 1786502525451671 us; error 39 us; skew 500 ppm
Aug 12 02:42:05 dist-test-slave-rmhg krb5kdc[8048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502525, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:42:05.454632 8067 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:42:05.455605 8067 webserver.cc:460] Webserver started at http://127.3.127.190:46019/ using document root <none> and password file <none>
I20260812 02:42:05.456161 8067 fs_manager.cc:359] Metadata directory not provided
I20260812 02:42:05.456223 8067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:42:05.456404 8067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:42:05.458676 8067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "a596a25ce239486d85cbde24e28227ba"
format_stamp: "Formatted at 2026-08-12 02:42:05 on dist-test-slave-rmhg"
server_key: "5953761930bc41f71b47bd14bdc65c3c"
server_key_iv: "be42b890c52978bfb4f7c96f2246abb9"
server_key_version: "encryptionkey@0"
I20260812 02:42:05.459267 8067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "a596a25ce239486d85cbde24e28227ba"
format_stamp: "Formatted at 2026-08-12 02:42:05 on dist-test-slave-rmhg"
server_key: "5953761930bc41f71b47bd14bdc65c3c"
server_key_iv: "be42b890c52978bfb4f7c96f2246abb9"
server_key_version: "encryptionkey@0"
I20260812 02:42:05.463439 8067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:42:05.466208 8089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:42:05.467572 8067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 02:42:05.467722 8067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "a596a25ce239486d85cbde24e28227ba"
format_stamp: "Formatted at 2026-08-12 02:42:05 on dist-test-slave-rmhg"
server_key: "5953761930bc41f71b47bd14bdc65c3c"
server_key_iv: "be42b890c52978bfb4f7c96f2246abb9"
server_key_version: "encryptionkey@0"
I20260812 02:42:05.467840 8067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:42:05.486157 8067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:42:05.489815 8067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:42:05.490025 8067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:42:05.497737 8067 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:40073
I20260812 02:42:05.498066 8184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:40073 every 8 connection(s)
I20260812 02:42:05.498858 8067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:42:05.502032 8185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:42:05.506582 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 8067
I20260812 02:42:05.506309 8185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40073 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:42:05.506734 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:42:05.507038 3582 external_mini_cluster.cc:1428] Setting key 73795c331a966bdd316d973e97ec7616
WARNING: no policy specified for kudu/127.3.127.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.189@KRBTEST.COM" created.
Aug 12 02:42:05 dist-test-slave-rmhg krb5kdc[8048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502525, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:42:05.525568 8189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.509452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.3.127.190:40073)
0812 02:42:05.510039 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:05.510061 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:05.510378 (+ 317us) server_negotiation.cc:407] Connection header received
0812 02:42:05.511693 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:05.511698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:05.511894 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:05.512082 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:05.513365 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.514383 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.515626 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.516034 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.516204 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:05.520667 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:05.520673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:05.520686 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:05.520712 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:05.522271 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:05.522758 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:05.522764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:05.522765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:05.522878 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:05.523187 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:05.523190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:05.523191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:05.523622 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:05.523752 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:05.524135 (+ 383us) server_negotiation.cc:299] Negotiation successful
0812 02:42:05.524377 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":179,"threads_started":1}
I20260812 02:42:05.525568 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.509452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:40073 (local address 127.0.0.1:37354)
0812 02:42:05.510198 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.510252 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 02:42:05.511313 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:42:05.512144 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:42:05.512150 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:42:05.512702 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:42:05.513120 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:05.513144 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.514531 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.514535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:05.515501 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:05.515512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.516204 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.516208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:05.516322 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:05.516963 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:42:05.516983 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:42:05.520466 (+ 3483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:42:05.522386 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:05.522396 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:05.522409 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:05.522640 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:05.522962 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:05.522965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:05.522967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:05.523086 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:05.523730 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:42:05.523736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:42:05.523884 (+ 148us) client_negotiation.cc:769] Sending connection context
0812 02:42:05.524037 (+ 153us) client_negotiation.cc:240] Negotiation successful
0812 02:42:05.524439 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":96,"spinlock_wait_cycles":59136,"thread_start_us":179,"threads_started":1}
I20260812 02:42:05.528441 8185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:42:05.529289 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.528928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:46427 (local address 127.0.0.1:36702)
0812 02:42:05.529083 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.529202 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.189:46427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260812 02:42:05.529490 8094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.127.189:46427: connect: Connection refused (error 111)
I20260812 02:42:05.530364 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.530159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:46427 (local address 127.0.0.1:36716)
0812 02:42:05.530256 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.530307 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.189:46427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260812 02:42:05.530493 8185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.189:46427: Network error: Client connection negotiation failed: client connection to 127.3.127.189:46427: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.189.keytab.
Entry for principal kudu/127.3.127.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.189.keytab.
I20260812 02:42:05.536301 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.189:46427
--webserver_interface=127.3.127.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.189.keytab
--principal=kudu/127.3.127.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:40073,127.3.127.189:46427,127.3.127.188:43211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:42:05.579771 8185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } attempt: 1
I20260812 02:42:05.580806 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.580437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:46427 (local address 127.0.0.1:36728)
0812 02:42:05.580635 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.580718 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.189:46427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 02:42:05.581764 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.581521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:46427 (local address 127.0.0.1:36736)
0812 02:42:05.581631 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.581693 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.189:46427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260812 02:42:05.581889 8185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.189:46427: Network error: Client connection negotiation failed: client connection to 127.3.127.189:46427: connect: Connection refused (error 111)
I20260812 02:42:05.650064 8185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } attempt: 2
I20260812 02:42:05.651017 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.650686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:46427 (local address 127.0.0.1:36738)
0812 02:42:05.650864 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.650933 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.189:46427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260812 02:42:05.651834 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.651612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:46427 (local address 127.0.0.1:36742)
0812 02:42:05.651724 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.651771 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.189:46427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260812 02:42:05.651952 8185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.189:46427: Network error: Client connection negotiation failed: client connection to 127.3.127.189:46427: connect: Connection refused (error 111)
W20260812 02:42:05.657645 8197 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 02:42:05.657955 8197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:42:05.658005 8197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:42:05.658072 8197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:42:05.662138 8197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:42:05.662199 8197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:42:05.662226 8197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:42:05.662257 8197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:42:05.662287 8197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:42:05.667778 8197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:40073,127.3.127.189:46427,127.3.127.188:43211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.189.keytab
--principal=kudu/127.3.127.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.189:46427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.127.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:42:05.668920 8197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:42:05.670047 8197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:42:05.678860 8208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:42:05.678869 8209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:42:05.678874 8213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:42:05.679253 8197 server_base.cc:1034] running on GCE node
I20260812 02:42:05.679862 8197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:42:05.681005 8197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:42:05.682207 8197 hybrid_clock.cc:648] HybridClock initialized: now 1786502525682180 us; error 40 us; skew 500 ppm
Aug 12 02:42:05 dist-test-slave-rmhg krb5kdc[8048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502525, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:42:05.685222 8197 init.cc:377] Logged in from keytab as kudu/127.3.127.189@KRBTEST.COM (short username kudu)
I20260812 02:42:05.686141 8197 webserver.cc:460] Webserver started at http://127.3.127.189:43411/ using document root <none> and password file <none>
I20260812 02:42:05.686719 8197 fs_manager.cc:359] Metadata directory not provided
I20260812 02:42:05.686781 8197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:42:05.686961 8197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:42:05.689059 8197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-1/data/instance:
uuid: "18b62278ddf7452486789fd5d518fd78"
format_stamp: "Formatted at 2026-08-12 02:42:05 on dist-test-slave-rmhg"
server_key: "6a6dd42cd0a9cdc93b97aab55e1067a8"
server_key_iv: "90f64aa3e3594ed7a11d74c5669c7114"
server_key_version: "encryptionkey@0"
I20260812 02:42:05.689671 8197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-1/wal/instance:
uuid: "18b62278ddf7452486789fd5d518fd78"
format_stamp: "Formatted at 2026-08-12 02:42:05 on dist-test-slave-rmhg"
server_key: "6a6dd42cd0a9cdc93b97aab55e1067a8"
server_key_iv: "90f64aa3e3594ed7a11d74c5669c7114"
server_key_version: "encryptionkey@0"
I20260812 02:42:05.693625 8197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:42:05.696081 8223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:42:05.697393 8197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 02:42:05.697546 8197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-1/wal
uuid: "18b62278ddf7452486789fd5d518fd78"
format_stamp: "Formatted at 2026-08-12 02:42:05 on dist-test-slave-rmhg"
server_key: "6a6dd42cd0a9cdc93b97aab55e1067a8"
server_key_iv: "90f64aa3e3594ed7a11d74c5669c7114"
server_key_version: "encryptionkey@0"
I20260812 02:42:05.697657 8197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:42:05.708925 8197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:42:05.712010 8197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:42:05.712198 8197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:42:05.719211 8197 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.189:46427
I20260812 02:42:05.719487 8303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.189:46427 every 8 connection(s)
I20260812 02:42:05.720217 8197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-1/data/info.pb
I20260812 02:42:05.723074 8304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:42:05.723950 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 8197
I20260812 02:42:05.724068 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-1/wal/instance
I20260812 02:42:05.724354 3582 external_mini_cluster.cc:1428] Setting key 4047fe06fa83e7e311bd809f743a4d82
I20260812 02:42:05.727264 8304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40073 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.127.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.188@KRBTEST.COM" created.
Aug 12 02:42:05 dist-test-slave-rmhg krb5kdc[8048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502525, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.189@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:42:05.741974 8189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.729269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37362 (local address 127.3.127.190:40073)
0812 02:42:05.729442 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:05.729446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:05.730285 (+ 839us) server_negotiation.cc:407] Connection header received
0812 02:42:05.731480 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:05.731483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:05.731535 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:05.731614 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:05.733247 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.733849 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.735250 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.735625 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.735720 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:05.738158 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:05.738161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:05.738163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:05.738188 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:05.739607 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:05.740154 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:05.740158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:05.740160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:05.740259 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:05.740657 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:05.740661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:05.740662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:05.740892 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:05.740996 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:05.741647 (+ 651us) server_negotiation.cc:299] Negotiation successful
0812 02:42:05.741803 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 02:42:05.742728 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.729608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:40073 (local address 127.0.0.1:37362)
0812 02:42:05.730115 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.730176 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 02:42:05.731247 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:42:05.731833 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:42:05.731843 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:42:05.732336 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:42:05.733030 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:05.733052 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.734013 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.734017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:05.735133 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:05.735144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.735788 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.735791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:05.735948 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:05.736505 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:42:05.736522 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:42:05.737991 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:42:05.739742 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:05.739747 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:05.739759 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:05.740040 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:05.740378 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:05.740381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:05.740383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:05.740542 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:05.741038 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:42:05.741043 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:42:05.741359 (+ 316us) client_negotiation.cc:769] Sending connection context
0812 02:42:05.741535 (+ 176us) client_negotiation.cc:240] Negotiation successful
0812 02:42:05.741844 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":117,"threads_started":1}
I20260812 02:42:05.743099 8185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } attempt: 3
I20260812 02:42:05.744171 8304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:42:05 dist-test-slave-rmhg krb5kdc[8048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502525, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for kudu/127.3.127.189@KRBTEST.COM
Aug 12 02:42:05 dist-test-slave-rmhg krb5kdc[8048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502525, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.189@KRBTEST.COM for kudu/127.3.127.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.188.keytab.
Entry for principal kudu/127.3.127.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.188.keytab.
I20260812 02:42:05.755944 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.188:43211
--webserver_interface=127.3.127.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.188.keytab
--principal=kudu/127.3.127.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:40073,127.3.127.189:46427,127.3.127.188:43211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:42:05.759864 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.743529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:46427 (local address 127.0.0.1:36746)
0812 02:42:05.743852 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.743867 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:42:05.743946 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:42:05.744780 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:42:05.744784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:42:05.744959 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:42:05.745095 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:05.745103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.746568 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.746572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:05.747719 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:05.747730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.748313 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.748317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:05.748426 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:05.748767 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:42:05.748779 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:42:05.750604 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:42:05.758315 (+ 7711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:05.758319 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:05.758321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:05.758568 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:05.758838 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:05.758841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:05.758843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:05.758930 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:05.759319 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:42:05.759322 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:42:05.759433 (+ 111us) client_negotiation.cc:769] Sending connection context
0812 02:42:05.759525 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 02:42:05.759684 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242}
I20260812 02:42:05.759881 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.744641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:46427 (local address 127.0.0.1:36760)
0812 02:42:05.744813 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.744826 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:42:05.744902 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:42:05.745190 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:42:05.745193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:42:05.745365 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:42:05.745455 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:05.745461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.746405 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.746408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:05.747215 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:05.747226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.748005 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.748010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:05.748164 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:05.748829 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:42:05.748845 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:42:05.751042 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:42:05.758116 (+ 7074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:05.758120 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:05.758122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:05.758351 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:05.758671 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:05.758674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:05.758676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:05.758788 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:05.759387 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:42:05.759390 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:42:05.759473 (+ 83us) client_negotiation.cc:769] Sending connection context
0812 02:42:05.759579 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 02:42:05.759716 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":1}
I20260812 02:42:05.759939 8311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.743832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36746 (local address 127.3.127.189:46427)
0812 02:42:05.744280 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:05.744291 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:05.744316 (+ 25us) server_negotiation.cc:407] Connection header received
0812 02:42:05.744360 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:05.744366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:05.744622 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:05.745255 (+ 633us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:05.745371 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.746435 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.747838 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.748197 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.748275 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:05.750743 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:05.750756 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:05.750775 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:05.750802 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:05.758162 (+ 7360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:05.758657 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:05.758661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:05.758663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:05.758751 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:05.759013 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:05.759017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:05.759019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:05.759221 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:05.759350 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:05.759630 (+ 280us) server_negotiation.cc:299] Negotiation successful
0812 02:42:05.759779 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":166,"server-negotiator.queue_time_us":338,"thread_start_us":122,"threads_started":1}
I20260812 02:42:05.760255 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.744690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36760 (local address 127.3.127.189:46427)
0812 02:42:05.744979 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:05.744988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:05.744999 (+ 11us) server_negotiation.cc:407] Connection header received
0812 02:42:05.745028 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:05.745030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:05.745061 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:05.745213 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:05.745563 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.746304 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.747392 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.747871 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.747974 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:05.751220 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:05.751224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:05.751227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:05.751254 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:05.757978 (+ 6724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:05.758463 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:05.758471 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:05.758473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:05.758577 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:05.758869 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:05.758872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:05.758873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:05.759234 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:05.759350 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:05.759996 (+ 646us) server_negotiation.cc:299] Negotiation successful
0812 02:42:05.760100 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"spinlock_wait_cycles":1792,"thread_start_us":96,"threads_started":1}
I20260812 02:42:05.762722 8304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:42:05.762868 8185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:42:05.763599 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.763294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43211 (local address 127.0.0.1:57598)
0812 02:42:05.763435 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.763525 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 02:42:05.763610 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.763313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43211 (local address 127.0.0.1:57614)
0812 02:42:05.763503 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.763548 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260812 02:42:05.764066 8228 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43211: connect: Connection refused (error 111)
I20260812 02:42:05.765062 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.764808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43211 (local address 127.0.0.1:57620)
0812 02:42:05.764944 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.764993 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 02:42:05.765398 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.765052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43211 (local address 127.0.0.1:57622)
0812 02:42:05.765277 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.765328 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
W20260812 02:42:05.765592 8185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.188:43211: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43211: connect: Connection refused (error 111)
W20260812 02:42:05.765635 8304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.188:43211: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43211: connect: Connection refused (error 111)
I20260812 02:42:05.796775 8185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } attempt: 1
I20260812 02:42:05.797737 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.797354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43211 (local address 127.0.0.1:57628)
0812 02:42:05.797548 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.797634 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 02:42:05.798574 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.798310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43211 (local address 127.0.0.1:57640)
0812 02:42:05.798431 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.798492 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 02:42:05.798724 8185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.188:43211: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43211: connect: Connection refused (error 111)
I20260812 02:42:05.814927 8304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } attempt: 1
I20260812 02:42:05.815878 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.815560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43211 (local address 127.0.0.1:57654)
0812 02:42:05.815722 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.815788 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 02:42:05.816619 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.816386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43211 (local address 127.0.0.1:57660)
0812 02:42:05.816507 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.816556 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260812 02:42:05.816744 8304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.188:43211: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43211: connect: Connection refused (error 111)
I20260812 02:42:05.873885 8185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } attempt: 2
I20260812 02:42:05.874790 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.874423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43211 (local address 127.0.0.1:57676)
0812 02:42:05.874606 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.874685 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 02:42:05.875630 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.875397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43211 (local address 127.0.0.1:57692)
0812 02:42:05.875500 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.875566 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260812 02:42:05.875766 8185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.188:43211: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43211: connect: Connection refused (error 111)
W20260812 02:42:05.877758 8313 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 02:42:05.878048 8313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:42:05.878090 8313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:42:05.878137 8313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:42:05.882254 8313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:42:05.882314 8313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:42:05.882331 8313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:42:05.882347 8313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:42:05.882364 8313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:42:05.884894 8304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } attempt: 2
I20260812 02:42:05.885689 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.885402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43211 (local address 127.0.0.1:57702)
0812 02:42:05.885542 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.885607 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260812 02:42:05.886392 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.886155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43211 (local address 127.0.0.1:57712)
0812 02:42:05.886261 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.886328 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260812 02:42:05.886525 8304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.188:43211: Network error: Client connection negotiation failed: client connection to 127.3.127.188:43211: connect: Connection refused (error 111)
I20260812 02:42:05.887997 8313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:40073,127.3.127.189:46427,127.3.127.188:43211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.188.keytab
--principal=kudu/127.3.127.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.188:43211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.127.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:42:05.889130 8313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:42:05.890272 8313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:42:05.899016 8326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:42:05.899016 8331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:42:05.899204 8313 server_base.cc:1034] running on GCE node
W20260812 02:42:05.899021 8324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:42:05.899906 8313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:42:05.901046 8313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:42:05.902254 8313 hybrid_clock.cc:648] HybridClock initialized: now 1786502525902226 us; error 43 us; skew 500 ppm
Aug 12 02:42:05 dist-test-slave-rmhg krb5kdc[8048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502525, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:42:05.905081 8313 init.cc:377] Logged in from keytab as kudu/127.3.127.188@KRBTEST.COM (short username kudu)
I20260812 02:42:05.906072 8313 webserver.cc:460] Webserver started at http://127.3.127.188:42683/ using document root <none> and password file <none>
I20260812 02:42:05.906642 8313 fs_manager.cc:359] Metadata directory not provided
I20260812 02:42:05.906714 8313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:42:05.906929 8313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:42:05.909010 8313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-2/data/instance:
uuid: "6d6f3036f5df4c9bbe63f024d4c6d667"
format_stamp: "Formatted at 2026-08-12 02:42:05 on dist-test-slave-rmhg"
server_key: "dc2c742457dced1c7c893c5cedc9e06f"
server_key_iv: "db37b6f43793cff52adcc651a0a10f85"
server_key_version: "encryptionkey@0"
I20260812 02:42:05.909605 8313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-2/wal/instance:
uuid: "6d6f3036f5df4c9bbe63f024d4c6d667"
format_stamp: "Formatted at 2026-08-12 02:42:05 on dist-test-slave-rmhg"
server_key: "dc2c742457dced1c7c893c5cedc9e06f"
server_key_iv: "db37b6f43793cff52adcc651a0a10f85"
server_key_version: "encryptionkey@0"
I20260812 02:42:05.913632 8313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:42:05.916263 8342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:42:05.917654 8313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:42:05.917783 8313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-2/wal
uuid: "6d6f3036f5df4c9bbe63f024d4c6d667"
format_stamp: "Formatted at 2026-08-12 02:42:05 on dist-test-slave-rmhg"
server_key: "dc2c742457dced1c7c893c5cedc9e06f"
server_key_iv: "db37b6f43793cff52adcc651a0a10f85"
server_key_version: "encryptionkey@0"
I20260812 02:42:05.917888 8313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:42:05.955057 8313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:42:05.958434 8313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:42:05.958630 8313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:42:05.966140 8313 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.188:43211
I20260812 02:42:05.966405 8419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.188:43211 every 8 connection(s)
I20260812 02:42:05.967190 8313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-2/data/info.pb
I20260812 02:42:05.970407 8420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:42:05.974675 8420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40073 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:42:05.974921 8185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } attempt: 3
I20260812 02:42:05.975595 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 8313
I20260812 02:42:05.975739 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786502224378336-3582-0/minicluster-data/master-2/wal/instance
I20260812 02:42:05.976030 3582 external_mini_cluster.cc:1428] Setting key f6065e0e7df6c73656a31676c7e3ca45
I20260812 02:42:05.977670 8304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } attempt: 3
Aug 12 02:42:05 dist-test-slave-rmhg krb5kdc[8048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502525, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for kudu/127.3.127.188@KRBTEST.COM
Aug 12 02:42:05 dist-test-slave-rmhg krb5kdc[8048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502525, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.189@KRBTEST.COM for kudu/127.3.127.188@KRBTEST.COM
Aug 12 02:42:05 dist-test-slave-rmhg krb5kdc[8048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.188@KRBTEST.COM
Aug 12 02:42:05 dist-test-slave-rmhg krb5kdc[8048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502525, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.188@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:42:05.990486 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.975406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43211 (local address 127.0.0.1:57720)
0812 02:42:05.975546 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.975562 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:42:05.975693 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:42:05.978665 (+ 2972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:42:05.978669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:42:05.978890 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:42:05.979012 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:05.979020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.981001 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.981005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:05.981890 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:05.981902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.982910 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.982914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:05.982996 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:05.983552 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:42:05.983568 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:42:05.985140 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:42:05.987897 (+ 2757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:05.987900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:05.987902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:05.988818 (+ 916us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:05.989136 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:05.989140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:05.989142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:05.989275 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:05.989965 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:42:05.989968 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:42:05.990072 (+ 104us) client_negotiation.cc:769] Sending connection context
0812 02:42:05.990160 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 02:42:05.990305 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 02:42:05.990486 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.978114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43211 (local address 127.0.0.1:57728)
0812 02:42:05.978394 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.978409 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:42:05.978495 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:42:05.978946 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:42:05.978948 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:42:05.979144 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:42:05.979252 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:05.979260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.980967 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.980970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:05.981886 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:05.981898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.982897 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.982901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:05.982987 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:05.984128 (+ 1141us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:42:05.984143 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:42:05.985710 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:42:05.988222 (+ 2512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:05.988226 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:05.988228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:05.988473 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:05.988802 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:05.988805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:05.988808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:05.988909 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:05.989965 (+ 1056us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:42:05.989968 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:42:05.990082 (+ 114us) client_negotiation.cc:769] Sending connection context
0812 02:42:05.990160 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 02:42:05.990306 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260812 02:42:05.990837 8189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.977102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37368 (local address 127.3.127.190:40073)
0812 02:42:05.977236 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:05.977240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:05.977400 (+ 160us) server_negotiation.cc:407] Connection header received
0812 02:42:05.978379 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:05.978383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:05.978431 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:05.978482 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:05.979749 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.980499 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.982192 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.982504 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.982630 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:05.987088 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:05.987092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:05.987094 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:05.987121 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:05.988580 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:05.989220 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:05.989229 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:05.989230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:05.989300 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:05.989642 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:05.989645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:05.989647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:05.989968 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:05.990083 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:05.990572 (+ 489us) server_negotiation.cc:299] Negotiation successful
0812 02:42:05.990696 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 02:42:05.992355 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.978155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57728 (local address 127.3.127.188:43211)
0812 02:42:05.978720 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:05.978729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:05.978740 (+ 11us) server_negotiation.cc:407] Connection header received
0812 02:42:05.978770 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:05.978773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:05.978809 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:05.979354 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:05.979463 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.980840 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.982035 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.982772 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.983483 (+ 711us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:05.985854 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:05.985857 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:05.985859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:05.985883 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:05.988112 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:05.988579 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:05.988584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:05.988590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:05.988700 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:05.989208 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:05.989211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:05.989213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:05.989835 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:05.990726 (+ 891us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:05.990865 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 02:42:05.991229 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":196,"server-negotiator.queue_time_us":506,"spinlock_wait_cycles":3968,"thread_start_us":97,"threads_started":1}
I20260812 02:42:05.992347 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.976732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:40073 (local address 127.0.0.1:37368)
0812 02:42:05.977253 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.977313 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 02:42:05.978115 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:42:05.978694 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:42:05.978699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:42:05.979138 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:42:05.979573 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:05.979590 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.980837 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.980840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:05.982070 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:05.982078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.982844 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.982848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:05.982965 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:05.983820 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:42:05.983842 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:42:05.986912 (+ 3070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:42:05.988661 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:05.988672 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:05.988679 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:05.988982 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:05.989397 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:05.989400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:05.989401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:05.989515 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:05.990146 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:42:05.990153 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:42:05.990317 (+ 164us) client_negotiation.cc:769] Sending connection context
0812 02:42:05.991629 (+ 1312us) client_negotiation.cc:240] Negotiation successful
0812 02:42:05.991825 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":66,"spinlock_wait_cycles":59648,"thread_start_us":127,"threads_started":1}
I20260812 02:42:05.992348 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.976492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57720 (local address 127.3.127.188:43211)
0812 02:42:05.977128 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:05.977146 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:05.977211 (+ 65us) server_negotiation.cc:407] Connection header received
0812 02:42:05.978112 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:05.978140 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:05.978455 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:05.979128 (+ 673us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:05.979283 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.980888 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.982045 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.982774 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.983251 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:05.985298 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:05.985304 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:05.985316 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:05.985339 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:05.987771 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:05.988919 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:05.988922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:05.988923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:05.989004 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:05.989390 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:05.989393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:05.989395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:05.989835 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:05.990894 (+ 1059us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:05.990964 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 02:42:05.991288 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":563,"server-negotiator.queue_time_us":491,"spinlock_wait_cycles":34048,"thread_start_us":161,"threads_started":1}
I20260812 02:42:05.992820 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.976670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57726 (local address 127.3.127.188:43211)
0812 02:42:05.977916 (+ 1246us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:05.977927 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:05.977944 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:42:05.978256 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:05.978260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:05.978454 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:05.978692 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:05.980075 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.980800 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.982051 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.982774 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.982934 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:05.986473 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:05.986477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:05.986479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:05.986504 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:05.990369 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:05.990907 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:05.990911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:05.990912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:05.990999 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:05.991313 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:05.991316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:05.991318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:05.991517 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:05.991672 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:05.992554 (+ 882us) server_negotiation.cc:299] Negotiation successful
0812 02:42:05.992672 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":432,"server-negotiator.queue_time_us":906,"spinlock_wait_cycles":12160,"thread_start_us":172,"threads_started":1}
I20260812 02:42:05.994434 8420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:42:05.997056 8185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba: Bootstrap starting.
I20260812 02:42:05.997577 8304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18b62278ddf7452486789fd5d518fd78: Bootstrap starting.
I20260812 02:42:05.999836 8304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18b62278ddf7452486789fd5d518fd78: Neither blocks nor log segments found. Creating new log.
I20260812 02:42:05.999842 8185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba: Neither blocks nor log segments found. Creating new log.
Aug 12 02:42:05 dist-test-slave-rmhg krb5kdc[8048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502525, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.188@KRBTEST.COM for kudu/127.3.127.189@KRBTEST.COM
I20260812 02:42:06.000640 8304 log.cc:824] T 00000000000000000000000000000000 P 18b62278ddf7452486789fd5d518fd78: Log is configured to *not* fsync() on all Append() calls
I20260812 02:42:06.000782 8185 log.cc:824] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba: Log is configured to *not* fsync() on all Append() calls
I20260812 02:42:06.003250 8304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18b62278ddf7452486789fd5d518fd78: No bootstrap required, opened a new log
I20260812 02:42:06.003995 8185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba: No bootstrap required, opened a new log
I20260812 02:42:06.004840 8311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.995334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36766 (local address 127.3.127.189:46427)
0812 02:42:05.995442 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:05.995446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:05.995516 (+ 70us) server_negotiation.cc:407] Connection header received
0812 02:42:05.995615 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:05.995618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:05.995664 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:05.995758 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:05.996205 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.996950 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.998130 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.998494 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.998609 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:06.000972 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:06.000976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:06.000978 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:06.001007 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:06.002783 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:06.003286 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:06.003290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:06.003291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:06.003426 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:06.003764 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:06.003767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:06.003769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:06.004029 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:06.004162 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:06.004558 (+ 396us) server_negotiation.cc:299] Negotiation successful
0812 02:42:06.004688 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 02:42:06.005472 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:05.995090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:46427 (local address 127.0.0.1:36766)
0812 02:42:05.995459 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:05.995471 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:42:05.995544 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:42:05.995787 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:42:05.995790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:42:05.995963 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:42:05.996084 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:05.996091 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.997084 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.997087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:05.998004 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:05.998016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:05.998623 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:05.998627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:05.998719 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:05.999221 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:42:05.999236 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:42:06.000796 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:42:06.002906 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:06.002909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:06.002911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:06.003166 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:06.003549 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:06.003552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:06.003554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:06.003652 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:06.004239 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:42:06.004242 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:42:06.004348 (+ 106us) client_negotiation.cc:769] Sending connection context
0812 02:42:06.005081 (+ 733us) client_negotiation.cc:240] Negotiation successful
0812 02:42:06.005304 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298}
I20260812 02:42:06.006520 8420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:42:06.007637 8304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18b62278ddf7452486789fd5d518fd78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a596a25ce239486d85cbde24e28227ba" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40073 } } peers { permanent_uuid: "18b62278ddf7452486789fd5d518fd78" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } } peers { permanent_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } }
I20260812 02:42:06.007899 8304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18b62278ddf7452486789fd5d518fd78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b62278ddf7452486789fd5d518fd78, State: Initialized, Role: FOLLOWER
I20260812 02:42:06.007993 8185 raft_consensus.cc:348] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a596a25ce239486d85cbde24e28227ba" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40073 } } peers { permanent_uuid: "18b62278ddf7452486789fd5d518fd78" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } } peers { permanent_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } }
I20260812 02:42:06.008219 8185 raft_consensus.cc:729] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a596a25ce239486d85cbde24e28227ba, State: Initialized, Role: FOLLOWER
I20260812 02:42:06.008265 8304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18b62278ddf7452486789fd5d518fd78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a596a25ce239486d85cbde24e28227ba" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40073 } } peers { permanent_uuid: "18b62278ddf7452486789fd5d518fd78" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } } peers { permanent_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } }
I20260812 02:42:06.008527 8185 consensus_queue.cc:260] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a596a25ce239486d85cbde24e28227ba" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40073 } } peers { permanent_uuid: "18b62278ddf7452486789fd5d518fd78" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } } peers { permanent_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } }
Aug 12 02:42:06 dist-test-slave-rmhg krb5kdc[8048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:42:06.009865 8304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18b62278ddf7452486789fd5d518fd78 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:42:06.009912 8185 sys_catalog.cc:564] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 02:42:06 dist-test-slave-rmhg krb5kdc[8048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502525, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.188@KRBTEST.COM for kudu/127.3.127.188@KRBTEST.COM
I20260812 02:42:06.014954 8189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:06.001293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37378 (local address 127.3.127.190:40073)
0812 02:42:06.001420 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:06.001424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:06.001436 (+ 12us) server_negotiation.cc:407] Connection header received
0812 02:42:06.001472 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:06.001475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:06.001521 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:06.001608 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:06.002560 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:06.003272 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:06.004530 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:06.004839 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:06.004912 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:06.009402 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:06.009407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:06.009409 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:06.009438 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:06.011387 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:06.011926 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:06.011931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:06.011932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:06.012032 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:06.012391 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:06.012395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:06.012396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:06.012641 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:06.012817 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:06.014683 (+ 1866us) server_negotiation.cc:299] Negotiation successful
0812 02:42:06.014808 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260812 02:42:06.016101 8478 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 18b62278ddf7452486789fd5d518fd78: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:42:06.016182 8478 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:42:06.016340 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:06.007288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57734 (local address 127.3.127.188:43211)
0812 02:42:06.007396 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:06.007401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:06.007415 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:42:06.007449 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:06.007452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:06.007504 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:06.007584 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:06.008132 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:06.008851 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:06.010123 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:06.010426 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:06.010498 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:06.012782 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:06.012786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:06.012788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:06.012811 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:06.014395 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:06.014844 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:06.014848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:06.014849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:06.014930 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:06.015434 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:06.015437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:06.015439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:06.015651 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:06.015751 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:06.016054 (+ 303us) server_negotiation.cc:299] Negotiation successful
0812 02:42:06.016194 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 02:42:06.017004 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:06.007039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:43211 (local address 127.0.0.1:57734)
0812 02:42:06.007287 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:06.007301 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:42:06.007372 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:42:06.007616 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:42:06.007619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:42:06.007781 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:42:06.007999 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:06.008009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:06.009129 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:06.009132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:06.009916 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:06.009928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:06.010632 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:06.010635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:06.010713 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:06.011110 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:42:06.011125 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:42:06.012650 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:42:06.014503 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:06.014506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:06.014508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:06.014747 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:06.015252 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:06.015255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:06.015257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:06.015345 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:06.015811 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:42:06.015814 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:42:06.015905 (+ 91us) client_negotiation.cc:769] Sending connection context
0812 02:42:06.016749 (+ 844us) client_negotiation.cc:240] Negotiation successful
0812 02:42:06.016889 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260812 02:42:06.020402 8420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d6f3036f5df4c9bbe63f024d4c6d667: Bootstrap starting.
I20260812 02:42:06.018899 8197 master_runner.cc:186] Error getting master registration for 127.3.127.188:43211: OK, instance_id { permanent_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667" instance_seqno: 1786502525963981 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 02:42:06.021502 8197 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 02:42:06.021313 8432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18b62278ddf7452486789fd5d518fd78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a596a25ce239486d85cbde24e28227ba" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40073 } } peers { permanent_uuid: "18b62278ddf7452486789fd5d518fd78" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } } peers { permanent_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } } }
I20260812 02:42:06.021579 8434 sys_catalog.cc:455] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a596a25ce239486d85cbde24e28227ba" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40073 } } peers { permanent_uuid: "18b62278ddf7452486789fd5d518fd78" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } } peers { permanent_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } } }
I20260812 02:42:06.021778 8432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18b62278ddf7452486789fd5d518fd78 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:42:06.021826 8434 sys_catalog.cc:458] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:42:06.023561 8420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d6f3036f5df4c9bbe63f024d4c6d667: Neither blocks nor log segments found. Creating new log.
I20260812 02:42:06.024427 8420 log.cc:824] T 00000000000000000000000000000000 P 6d6f3036f5df4c9bbe63f024d4c6d667: Log is configured to *not* fsync() on all Append() calls
I20260812 02:42:06.022107 8067 master_runner.cc:186] Error getting master registration for 127.3.127.188:43211: OK, instance_id { permanent_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667" instance_seqno: 1786502525963981 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260812 02:42:06.025315 8467 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260812 02:42:06.025398 8067 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 02:42:06.025393 8467 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:42:06.031483 8420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d6f3036f5df4c9bbe63f024d4c6d667: No bootstrap required, opened a new log
Aug 12 02:42:06 dist-test-slave-rmhg krb5kdc[8048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.189@KRBTEST.COM
I20260812 02:42:06.037056 8420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d6f3036f5df4c9bbe63f024d4c6d667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a596a25ce239486d85cbde24e28227ba" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40073 } } peers { permanent_uuid: "18b62278ddf7452486789fd5d518fd78" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } } peers { permanent_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } }
I20260812 02:42:06.037364 8420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d6f3036f5df4c9bbe63f024d4c6d667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d6f3036f5df4c9bbe63f024d4c6d667, State: Initialized, Role: FOLLOWER
I20260812 02:42:06.037741 8420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d6f3036f5df4c9bbe63f024d4c6d667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a596a25ce239486d85cbde24e28227ba" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40073 } } peers { permanent_uuid: "18b62278ddf7452486789fd5d518fd78" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } } peers { permanent_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } }
I20260812 02:42:06.038127 8311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:06.017666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36772 (local address 127.3.127.189:46427)
0812 02:42:06.021602 (+ 3936us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:06.021606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:06.025109 (+ 3503us) server_negotiation.cc:407] Connection header received
0812 02:42:06.025333 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:06.025337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:06.025389 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:06.025455 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:06.026431 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:06.027221 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:06.028393 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:06.028755 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:06.028832 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:06.034028 (+ 5196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:06.034033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:06.034036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:06.034068 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:06.035838 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:06.036468 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:06.036472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:06.036475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:06.036582 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:06.036884 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:06.036887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:06.036889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:06.037119 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:06.037330 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:06.037819 (+ 489us) server_negotiation.cc:299] Negotiation successful
0812 02:42:06.037956 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3853}
I20260812 02:42:06.039064 8484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d6f3036f5df4c9bbe63f024d4c6d667 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a596a25ce239486d85cbde24e28227ba" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40073 } } peers { permanent_uuid: "18b62278ddf7452486789fd5d518fd78" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } } peers { permanent_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } } }
I20260812 02:42:06.039283 8484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d6f3036f5df4c9bbe63f024d4c6d667 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:42:06.039858 8420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d6f3036f5df4c9bbe63f024d4c6d667 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:42:06.042021 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:42:06.043457 8137 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a596a25ce239486d85cbde24e28227ba"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37378
I20260812 02:42:06.043814 8137 raft_consensus.cc:482] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 02:42:06.043967 8137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:42:06.045454 8137 raft_consensus.cc:504] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a596a25ce239486d85cbde24e28227ba" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40073 } } peers { permanent_uuid: "18b62278ddf7452486789fd5d518fd78" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } } peers { permanent_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } }
W20260812 02:42:06.046337 8508 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d6f3036f5df4c9bbe63f024d4c6d667: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:42:06.046460 8508 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:42:06.046936 8264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a596a25ce239486d85cbde24e28227ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "18b62278ddf7452486789fd5d518fd78"
I20260812 02:42:06.047202 8264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18b62278ddf7452486789fd5d518fd78 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:42:06.048564 8264 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 18b62278ddf7452486789fd5d518fd78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a596a25ce239486d85cbde24e28227ba in term 1.
I20260812 02:42:06.048998 8094 leader_election.cc:302] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b62278ddf7452486789fd5d518fd78, a596a25ce239486d85cbde24e28227ba; no voters:
I20260812 02:42:06.049362 8434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:42:06.049587 8434 raft_consensus.cc:686] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [term 1 LEADER]: Becoming Leader. State: Replica: a596a25ce239486d85cbde24e28227ba, State: Running, Role: LEADER
I20260812 02:42:06.049937 8434 consensus_queue.cc:237] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a596a25ce239486d85cbde24e28227ba" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40073 } } peers { permanent_uuid: "18b62278ddf7452486789fd5d518fd78" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } } peers { permanent_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } }
I20260812 02:42:06.051779 8434 sys_catalog.cc:455] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader a596a25ce239486d85cbde24e28227ba. Latest consensus state: current_term: 1 leader_uuid: "a596a25ce239486d85cbde24e28227ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a596a25ce239486d85cbde24e28227ba" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40073 } } peers { permanent_uuid: "18b62278ddf7452486789fd5d518fd78" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } } peers { permanent_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } } }
I20260812 02:42:06.051893 8434 sys_catalog.cc:458] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [sys.catalog]: This master's current role is: LEADER
I20260812 02:42:06.054128 8313 master_runner.cc:263] No leader master found from master 6d6f3036f5df4c9bbe63f024d4c6d667
I20260812 02:42:06.054199 8313 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 02:42:06.054358 8137 leader_election.cc:288] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [CANDIDATE]: Term 1 election: Requested vote from peers 18b62278ddf7452486789fd5d518fd78 (127.3.127.189:46427), 6d6f3036f5df4c9bbe63f024d4c6d667 (127.3.127.188:43211)
I20260812 02:42:06.056918 8386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a596a25ce239486d85cbde24e28227ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667"
I20260812 02:42:06.057214 8386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d6f3036f5df4c9bbe63f024d4c6d667 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:42:06.058588 8386 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6d6f3036f5df4c9bbe63f024d4c6d667 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a596a25ce239486d85cbde24e28227ba in term 1.
I20260812 02:42:06.058952 8514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:42:06.059957 8514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:42:06.075659 8386 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6d6f3036f5df4c9bbe63f024d4c6d667 [term 1 FOLLOWER]: Refusing update from remote peer a596a25ce239486d85cbde24e28227ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:42:06.076606 8264 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 18b62278ddf7452486789fd5d518fd78 [term 1 FOLLOWER]: Refusing update from remote peer a596a25ce239486d85cbde24e28227ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:42:06.077554 8511 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "18b62278ddf7452486789fd5d518fd78" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:42:06.078649 8511 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:42:06.083015 8432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18b62278ddf7452486789fd5d518fd78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a596a25ce239486d85cbde24e28227ba. Latest consensus state: current_term: 1 leader_uuid: "a596a25ce239486d85cbde24e28227ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a596a25ce239486d85cbde24e28227ba" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40073 } } peers { permanent_uuid: "18b62278ddf7452486789fd5d518fd78" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } } peers { permanent_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } } }
I20260812 02:42:06.083169 8432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18b62278ddf7452486789fd5d518fd78 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:42:06.086786 8484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d6f3036f5df4c9bbe63f024d4c6d667 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a596a25ce239486d85cbde24e28227ba. Latest consensus state: current_term: 1 leader_uuid: "a596a25ce239486d85cbde24e28227ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a596a25ce239486d85cbde24e28227ba" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40073 } } peers { permanent_uuid: "18b62278ddf7452486789fd5d518fd78" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } } peers { permanent_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } } }
I20260812 02:42:06.086918 8484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d6f3036f5df4c9bbe63f024d4c6d667 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:42:06.095078 8511 sys_catalog.cc:455] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a596a25ce239486d85cbde24e28227ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a596a25ce239486d85cbde24e28227ba" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40073 } } peers { permanent_uuid: "18b62278ddf7452486789fd5d518fd78" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } } peers { permanent_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } } }
I20260812 02:42:06.095247 8511 sys_catalog.cc:458] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [sys.catalog]: This master's current role is: LEADER
I20260812 02:42:06.096576 8432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18b62278ddf7452486789fd5d518fd78 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a596a25ce239486d85cbde24e28227ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a596a25ce239486d85cbde24e28227ba" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40073 } } peers { permanent_uuid: "18b62278ddf7452486789fd5d518fd78" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } } peers { permanent_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } } }
I20260812 02:42:06.096714 8432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18b62278ddf7452486789fd5d518fd78 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:42:06.097172 8524 mvcc.cc:205] Tried to move back new op lower bound from 7317514346796769280 to 7317514346704175104. Current Snapshot: MvccSnapshot[applied={T|T < 7317514346796769280}]
I20260812 02:42:06.098948 8311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:06.070505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.3.127.189:46427)
0812 02:42:06.070631 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:06.070635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:06.081738 (+ 11103us) server_negotiation.cc:407] Connection header received
0812 02:42:06.083258 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:06.083261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:06.083315 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:06.083364 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:06.084279 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:06.085113 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:06.087237 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:06.087541 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:06.087636 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:06.090490 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:06.090494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:06.090497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:06.090525 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:06.092075 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:06.094877 (+ 2802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:06.094881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:06.094883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:06.095010 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:06.097275 (+ 2265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:06.097279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:06.097280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:06.097524 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:06.097643 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:06.098660 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0812 02:42:06.098795 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 02:42:06.100775 8530 mvcc.cc:205] Tried to move back new op lower bound from 7317514346796769280 to 7317514346704175104. Current Snapshot: MvccSnapshot[applied={T|T < 7317514346796769280}]
I20260812 02:42:06.100850 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:06.070789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57744 (local address 127.3.127.188:43211)
0812 02:42:06.070922 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:06.070927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:06.085428 (+ 14501us) server_negotiation.cc:407] Connection header received
0812 02:42:06.085616 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:06.085620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:06.085671 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:06.085726 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:06.086640 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:06.087318 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:06.088304 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:06.088636 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:06.088714 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:06.092598 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:06.092602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:06.092604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:06.092634 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:06.094237 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:06.095939 (+ 1702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:06.095944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:06.095945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:06.096054 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:06.097953 (+ 1899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:06.097957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:06.097958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:06.098200 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:06.098298 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:06.100571 (+ 2273us) server_negotiation.cc:299] Negotiation successful
0812 02:42:06.100700 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 02:42:06.101681 8434 sys_catalog.cc:455] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a596a25ce239486d85cbde24e28227ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a596a25ce239486d85cbde24e28227ba" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40073 } } peers { permanent_uuid: "18b62278ddf7452486789fd5d518fd78" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } } peers { permanent_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } } }
I20260812 02:42:06.101817 8434 sys_catalog.cc:458] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba [sys.catalog]: This master's current role is: LEADER
I20260812 02:42:06.103058 8484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d6f3036f5df4c9bbe63f024d4c6d667 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a596a25ce239486d85cbde24e28227ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a596a25ce239486d85cbde24e28227ba" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 40073 } } peers { permanent_uuid: "18b62278ddf7452486789fd5d518fd78" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46427 } } peers { permanent_uuid: "6d6f3036f5df4c9bbe63f024d4c6d667" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 43211 } } }
I20260812 02:42:06.103185 8484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d6f3036f5df4c9bbe63f024d4c6d667 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:42:06.103718 8535 mvcc.cc:205] Tried to move back new op lower bound from 7317514346796769280 to 7317514346704175104. Current Snapshot: MvccSnapshot[applied={T|T < 7317514346796769280}]
I20260812 02:42:06.111338 8189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:06.069708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37390 (local address 127.3.127.190:40073)
0812 02:42:06.069872 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:06.069877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:06.081516 (+ 11639us) server_negotiation.cc:407] Connection header received
0812 02:42:06.082052 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:06.082056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:06.082111 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:06.082163 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:06.086277 (+ 4114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:06.087006 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:06.102687 (+ 15681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:06.103039 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:06.103122 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:06.106427 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:06.106432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:06.106435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:06.106468 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:06.108106 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:06.109872 (+ 1766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:06.109876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:06.109878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:06.109971 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:06.110294 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:06.110297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:06.110299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:06.110535 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:06.110645 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:06.111039 (+ 394us) server_negotiation.cc:299] Negotiation successful
0812 02:42:06.111173 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 02:42:06.114342 8514 catalog_manager.cc:1347] Generated new cluster ID: 5f76612240db4647ba22175466441b55
I20260812 02:42:06.114425 8514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:42:06.146214 8514 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:42:06.147523 8514 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:42:06.167434 8514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a596a25ce239486d85cbde24e28227ba: Generated new TSK 0
I20260812 02:42:06.168268 8514 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 02:42:06.207917 8189 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:06.204175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37406 (local address 127.3.127.190:40073)
0812 02:42:06.204340 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:06.204344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:06.206642 (+ 2298us) server_negotiation.cc:407] Connection header received
0812 02:42:06.206886 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:06.206891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:06.206951 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:06.207014 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:06.207760 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37406: BlockingRecv error: recv got EOF from 127.0.0.1:37406 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260812 02:42:06.209525 8311 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:06.205105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36796 (local address 127.3.127.189:46427)
0812 02:42:06.205252 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:06.205257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:06.208502 (+ 3245us) server_negotiation.cc:407] Connection header received
0812 02:42:06.208691 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:06.208694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:06.208746 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:06.208802 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:06.209380 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36796: BlockingRecv error: recv got EOF from 127.0.0.1:36796 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260812 02:42:06.212409 8423 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:06.205858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57758 (local address 127.3.127.188:43211)
0812 02:42:06.205979 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:06.205984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:06.211322 (+ 5338us) server_negotiation.cc:407] Connection header received
0812 02:42:06.211534 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:06.211538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:06.211596 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:06.211655 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:06.212271 (+ 616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57758: BlockingRecv error: recv got EOF from 127.0.0.1:57758 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260812 02:42:06.218816 8311 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:06.216899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36802 (local address 127.3.127.189:46427)
0812 02:42:06.217026 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:06.217031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:06.217683 (+ 652us) server_negotiation.cc:407] Connection header received
0812 02:42:06.217934 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:06.217938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:06.218025 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:06.218084 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:06.218691 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36802: BlockingRecv error: recv got EOF from 127.0.0.1:36802 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260812 02:42:06.224560 8311 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:06.222841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.3.127.189:46427)
0812 02:42:06.222957 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:06.222961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:06.223517 (+ 556us) server_negotiation.cc:407] Connection header received
0812 02:42:06.223726 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:06.223730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:06.223787 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:06.223838 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:06.224402 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36812: BlockingRecv error: recv got EOF from 127.0.0.1:36812 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260812 02:42:06.230556 8311 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:06.229221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36824 (local address 127.3.127.189:46427)
0812 02:42:06.229354 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:06.229358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:06.229601 (+ 243us) server_negotiation.cc:407] Connection header received
0812 02:42:06.229754 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:06.229757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:06.229815 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:06.229907 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:06.230415 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36824: BlockingRecv error: recv got EOF from 127.0.0.1:36824 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260812 02:42:06.238366 8311 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:06.237006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36834 (local address 127.3.127.189:46427)
0812 02:42:06.237198 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:06.237203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:06.237428 (+ 225us) server_negotiation.cc:407] Connection header received
0812 02:42:06.237622 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:06.237625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:06.237686 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:06.237782 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:06.238236 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36834: BlockingRecv error: recv got EOF from 127.0.0.1:36834 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260812 02:42:06.249765 8311 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:06.248545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.3.127.189:46427)
0812 02:42:06.248695 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:06.248700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:06.248838 (+ 138us) server_negotiation.cc:407] Connection header received
0812 02:42:06.248997 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:06.249001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:06.249055 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:06.249127 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:06.249640 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36842: BlockingRecv error: recv got EOF from 127.0.0.1:36842 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260812 02:42:06.269109 8311 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:06.267954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36850 (local address 127.3.127.189:46427)
0812 02:42:06.268095 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:06.268099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:06.268220 (+ 121us) server_negotiation.cc:407] Connection header received
0812 02:42:06.268353 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:06.268356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:06.268408 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:06.268493 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:06.269003 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36850: BlockingRecv error: recv got EOF from 127.0.0.1:36850 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260812 02:42:06.304687 8311 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:06.303585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36852 (local address 127.3.127.189:46427)
0812 02:42:06.303738 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:06.303741 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:06.303870 (+ 129us) server_negotiation.cc:407] Connection header received
0812 02:42:06.304002 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:06.304005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:06.304057 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:06.304148 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:06.304578 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36852: BlockingRecv error: recv got EOF from 127.0.0.1:36852 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260812 02:42:06.372370 8311 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:06.371058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36854 (local address 127.3.127.189:46427)
0812 02:42:06.371237 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:06.371242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:06.371403 (+ 161us) server_negotiation.cc:407] Connection header received
0812 02:42:06.371568 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:06.371570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:06.371643 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:06.371716 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:06.372260 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36854: BlockingRecv error: recv got EOF from 127.0.0.1:36854 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260812 02:42:06.504256 8311 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:06.502915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36856 (local address 127.3.127.189:46427)
0812 02:42:06.503094 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:06.503099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:06.503294 (+ 195us) server_negotiation.cc:407] Connection header received
0812 02:42:06.503426 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:06.503429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:06.503485 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:06.503566 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:06.504132 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36856: BlockingRecv error: recv got EOF from 127.0.0.1:36856 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260812 02:42:06.763813 8311 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:06.762784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36860 (local address 127.3.127.189:46427)
0812 02:42:06.762919 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:06.762922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:06.762938 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:42:06.763054 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:06.763056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:06.763109 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:06.763194 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:06.763702 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36860: BlockingRecv error: recv got EOF from 127.0.0.1:36860 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 02:42:07.017652 8478 catalog_manager.cc:1259] Loaded cluster ID: 5f76612240db4647ba22175466441b55
I20260812 02:42:07.017745 8478 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 18b62278ddf7452486789fd5d518fd78: loading cluster ID for follower catalog manager: success
I20260812 02:42:07.019482 8478 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 18b62278ddf7452486789fd5d518fd78: acquiring CA information for follower catalog manager: success
I20260812 02:42:07.020735 8478 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 18b62278ddf7452486789fd5d518fd78: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:42:07.047901 8508 catalog_manager.cc:1259] Loaded cluster ID: 5f76612240db4647ba22175466441b55
I20260812 02:42:07.047991 8508 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6d6f3036f5df4c9bbe63f024d4c6d667: loading cluster ID for follower catalog manager: success
I20260812 02:42:07.049844 8508 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6d6f3036f5df4c9bbe63f024d4c6d667: acquiring CA information for follower catalog manager: success
I20260812 02:42:07.051144 8508 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6d6f3036f5df4c9bbe63f024d4c6d667: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:42:07.282004 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:07.278115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.3.127.189:46427)
0812 02:42:07.278395 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:07.278398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:07.278415 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:42:07.278521 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:07.278523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:07.278656 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:07.278773 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:42:07.279001 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:07.279531 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:07.280421 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:07.280647 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:07.280729 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:07.281078 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:42:07.281595 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:42:07.281713 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:07.281775 (+ 62us) server_negotiation.cc:299] Negotiation successful
0812 02:42:07.281837 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":93,"threads_started":1}
I20260812 02:42:07.288987 8666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:07.285280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57768 (local address 127.3.127.188:43211)
0812 02:42:07.285509 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:07.285513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:07.285527 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:42:07.285705 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:07.285708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:07.285833 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:07.285929 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:42:07.286183 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:07.286709 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:07.287558 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:07.287795 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:07.287870 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:07.288215 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:42:07.288654 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:42:07.288742 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:07.288805 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 02:42:07.288858 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
I20260812 02:42:07.296675 8676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:07.291996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37416 (local address 127.3.127.190:40073)
0812 02:42:07.292275 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:07.292280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:07.292298 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:42:07.292403 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:07.292406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:07.292602 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:07.292736 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:42:07.292963 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:07.293723 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:07.294806 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:07.295040 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:07.295133 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:07.295895 (+ 762us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:42:07.296275 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:42:07.296358 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:07.296434 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 02:42:07.296503 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":90,"threads_started":1}
I20260812 02:42:07.296662 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:07.292929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.3.127.189:46427)
0812 02:42:07.293085 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:07.293090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:07.293106 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:42:07.293188 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:07.293191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:07.293244 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:07.293310 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:42:07.293553 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:07.294275 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:07.295442 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:07.295676 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:07.295753 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:07.296015 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:42:07.296095 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:42:07.296173 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:07.296496 (+ 323us) server_negotiation.cc:299] Negotiation successful
0812 02:42:07.296552 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 02:42:07.296855 8666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:07.292746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57774 (local address 127.3.127.188:43211)
0812 02:42:07.292865 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:07.292869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:07.293182 (+ 313us) server_negotiation.cc:407] Connection header received
0812 02:42:07.293305 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:07.293308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:07.293359 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:07.293439 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:42:07.293638 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:07.294185 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:07.295640 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:07.295923 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:07.296016 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:07.296307 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:42:07.296387 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:42:07.296448 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:07.296664 (+ 216us) server_negotiation.cc:299] Negotiation successful
0812 02:42:07.296720 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 02:42:07.299964 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 8067
I20260812 02:42:07.314292 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 8197
I20260812 02:42:07.329754 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 8313
2026-08-12T02:42:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3326 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:42:07 dist-test-slave-rmhg krb5kdc[8687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:42:07 dist-test-slave-rmhg krb5kdc[8687](info): set up 2 sockets
Aug 12 02:42:07 dist-test-slave-rmhg krb5kdc[8687](info): commencing operation
krb5kdc: starting...
W20260812 02:42:08.499401 3582 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/test-user.keytab.
Aug 12 02:42:08 dist-test-slave-rmhg krb5kdc[8687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:42:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:42:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
Entry for principal kudu/127.3.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab.
I20260812 02:42:08.630138 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:44455
--webserver_interface=127.3.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:44457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:44455,127.3.127.189:46567,127.3.127.188:45891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:42:08.745945 8711 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 02:42:08.746270 8711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:42:08.746294 8711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:42:08.746345 8711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:42:08.750445 8711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:42:08.750521 8711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:42:08.750546 8711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:42:08.750563 8711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:42:08.750577 8711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:42:08.755978 8711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:44457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:44455,127.3.127.189:46567,127.3.127.188:45891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.190.keytab
--principal=kudu/127.3.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.190:44455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:42:08.757077 8711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:42:08.758141 8711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:42:08.766862 8721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:42:08.766862 8720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:42:08.766873 8723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:42:08.767002 8711 server_base.cc:1034] running on GCE node
I20260812 02:42:08.767786 8711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:42:08.768947 8711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:42:08.770165 8711 hybrid_clock.cc:648] HybridClock initialized: now 1786502528770161 us; error 42 us; skew 500 ppm
Aug 12 02:42:08 dist-test-slave-rmhg krb5kdc[8687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502528, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:42:08.772993 8711 init.cc:377] Logged in from keytab as kudu/127.3.127.190@KRBTEST.COM (short username kudu)
I20260812 02:42:08.773948 8711 webserver.cc:460] Webserver started at http://127.3.127.190:45603/ using document root <none> and password file <none>
I20260812 02:42:08.774492 8711 fs_manager.cc:359] Metadata directory not provided
I20260812 02:42:08.774609 8711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:42:08.774829 8711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:42:08.777009 8711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-0/data/instance:
uuid: "bd0d6199a40d4f969f124ef417b356a8"
format_stamp: "Formatted at 2026-08-12 02:42:08 on dist-test-slave-rmhg"
server_key: "c32d575c25c8c7cfdf3de6daf058d71c"
server_key_iv: "5678b01030ae87d0117206120f6bdc55"
server_key_version: "encryptionkey@0"
I20260812 02:42:08.777657 8711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-0/wal/instance:
uuid: "bd0d6199a40d4f969f124ef417b356a8"
format_stamp: "Formatted at 2026-08-12 02:42:08 on dist-test-slave-rmhg"
server_key: "c32d575c25c8c7cfdf3de6daf058d71c"
server_key_iv: "5678b01030ae87d0117206120f6bdc55"
server_key_version: "encryptionkey@0"
I20260812 02:42:08.781626 8711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:42:08.784201 8737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:42:08.785564 8711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:42:08.785696 8711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-0/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-0/wal
uuid: "bd0d6199a40d4f969f124ef417b356a8"
format_stamp: "Formatted at 2026-08-12 02:42:08 on dist-test-slave-rmhg"
server_key: "c32d575c25c8c7cfdf3de6daf058d71c"
server_key_iv: "5678b01030ae87d0117206120f6bdc55"
server_key_version: "encryptionkey@0"
I20260812 02:42:08.785796 8711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:42:08.795797 8711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:42:08.799315 8711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:42:08.799525 8711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:42:08.807180 8711 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.190:44455
I20260812 02:42:08.807557 8829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.190:44455 every 8 connection(s)
I20260812 02:42:08.808277 8711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-0/data/info.pb
I20260812 02:42:08.811374 8831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:42:08.815455 8831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.190" port: 44455 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:42:08.817596 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 8711
I20260812 02:42:08.817727 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-0/wal/instance
I20260812 02:42:08.818037 3582 external_mini_cluster.cc:1428] Setting key e9077d760fe2ede5f517ccf0da72fd36
Aug 12 02:42:08 dist-test-slave-rmhg krb5kdc[8687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502528, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.127.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.189@KRBTEST.COM" created.
I20260812 02:42:08.831276 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:08.817966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:44455 (local address 127.0.0.1:33020)
0812 02:42:08.818540 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:08.818595 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 02:42:08.819469 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:42:08.820257 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:42:08.820262 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:42:08.820623 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:42:08.820917 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:08.820932 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:08.822298 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:08.822301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:08.823040 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:08.823059 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:08.823703 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:08.823707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:08.823818 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:08.824376 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:42:08.824395 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:42:08.826114 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:42:08.828075 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:08.828084 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:08.828095 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:08.828308 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:08.828652 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:08.828655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:08.828657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:08.828795 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:08.829485 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:42:08.829492 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:42:08.829640 (+ 148us) client_negotiation.cc:769] Sending connection context
0812 02:42:08.829801 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 02:42:08.830223 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":120,"thread_start_us":164,"threads_started":1}
I20260812 02:42:08.831276 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:08.817966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.3.127.190:44455)
0812 02:42:08.818557 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:08.818576 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:08.818703 (+ 127us) server_negotiation.cc:407] Connection header received
0812 02:42:08.819779 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:08.819784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:08.820030 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:08.820226 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:08.821085 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:08.822176 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:08.823166 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:08.823549 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:08.823710 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:08.826262 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:08.826269 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:08.826281 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:08.826303 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:08.827966 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:08.828412 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:08.828418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:08.828419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:08.828524 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:08.828879 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:08.828882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:08.828884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:08.829381 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:08.829509 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:08.829876 (+ 367us) server_negotiation.cc:299] Negotiation successful
0812 02:42:08.830168 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":473,"spinlock_wait_cycles":56704,"thread_start_us":189,"threads_started":1}
I20260812 02:42:08.834738 8831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:42:08.835638 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:08.835280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:46567 (local address 127.0.0.1:51524)
0812 02:42:08.835426 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:08.835519 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.189:46567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 02:42:08.835862 8747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.127.189:46567: connect: Connection refused (error 111)
I20260812 02:42:08.836830 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:08.836589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:46567 (local address 127.0.0.1:51538)
0812 02:42:08.836701 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:08.836758 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.189:46567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260812 02:42:08.837031 8831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.189:46567: Network error: Client connection negotiation failed: client connection to 127.3.127.189:46567: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.189.keytab.
Entry for principal kudu/127.3.127.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.189.keytab.
I20260812 02:42:08.851089 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.189:46567
--webserver_interface=127.3.127.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.189.keytab
--principal=kudu/127.3.127.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:44457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:44455,127.3.127.189:46567,127.3.127.188:45891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:42:08.886315 8831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } attempt: 1
I20260812 02:42:08.887248 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:08.886906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:46567 (local address 127.0.0.1:51554)
0812 02:42:08.887078 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:08.887146 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.189:46567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 02:42:08.887930 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:08.887746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:46567 (local address 127.0.0.1:51560)
0812 02:42:08.887830 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:08.887872 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.189:46567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260812 02:42:08.888047 8831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.189:46567: Network error: Client connection negotiation failed: client connection to 127.3.127.189:46567: connect: Connection refused (error 111)
I20260812 02:42:08.956236 8831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } attempt: 2
I20260812 02:42:08.957079 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:08.956786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:46567 (local address 127.0.0.1:51566)
0812 02:42:08.956933 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:08.956999 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.189:46567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 02:42:08.957885 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:08.957662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:46567 (local address 127.0.0.1:51572)
0812 02:42:08.957771 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:08.957820 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.189:46567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 02:42:08.958005 8831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.189:46567: Network error: Client connection negotiation failed: client connection to 127.3.127.189:46567: connect: Connection refused (error 111)
W20260812 02:42:08.967504 8848 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 02:42:08.967800 8848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:42:08.967828 8848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:42:08.967891 8848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:42:08.971998 8848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:42:08.972064 8848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:42:08.972081 8848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:42:08.972097 8848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:42:08.972115 8848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:42:08.977571 8848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:44457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:44455,127.3.127.189:46567,127.3.127.188:45891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.189.keytab
--principal=kudu/127.3.127.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.189:46567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.127.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:42:08.978724 8848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:42:08.979769 8848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:42:08.988675 8848 server_base.cc:1034] running on GCE node
W20260812 02:42:08.988581 8859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:42:08.988586 8860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:42:08.988613 8863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:42:08.989607 8848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:42:08.990820 8848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:42:08.991986 8848 hybrid_clock.cc:648] HybridClock initialized: now 1786502528991966 us; error 30 us; skew 500 ppm
Aug 12 02:42:08 dist-test-slave-rmhg krb5kdc[8687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502528, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:42:08.994900 8848 init.cc:377] Logged in from keytab as kudu/127.3.127.189@KRBTEST.COM (short username kudu)
I20260812 02:42:08.995878 8848 webserver.cc:460] Webserver started at http://127.3.127.189:37585/ using document root <none> and password file <none>
I20260812 02:42:08.996461 8848 fs_manager.cc:359] Metadata directory not provided
I20260812 02:42:08.996531 8848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:42:08.996769 8848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:42:08.998948 8848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-1/data/instance:
uuid: "77a8c0c3fb4a43c58a30ba4e53214643"
format_stamp: "Formatted at 2026-08-12 02:42:08 on dist-test-slave-rmhg"
server_key: "43533c8e9a6239c952e6629070304c6c"
server_key_iv: "f95fe9a1d122d4135d2f396bb721676e"
server_key_version: "encryptionkey@0"
I20260812 02:42:08.999524 8848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-1/wal/instance:
uuid: "77a8c0c3fb4a43c58a30ba4e53214643"
format_stamp: "Formatted at 2026-08-12 02:42:08 on dist-test-slave-rmhg"
server_key: "43533c8e9a6239c952e6629070304c6c"
server_key_iv: "f95fe9a1d122d4135d2f396bb721676e"
server_key_version: "encryptionkey@0"
I20260812 02:42:09.003520 8848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260812 02:42:09.006242 8877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:42:09.007529 8848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 02:42:09.007659 8848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-1/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-1/wal
uuid: "77a8c0c3fb4a43c58a30ba4e53214643"
format_stamp: "Formatted at 2026-08-12 02:42:08 on dist-test-slave-rmhg"
server_key: "43533c8e9a6239c952e6629070304c6c"
server_key_iv: "f95fe9a1d122d4135d2f396bb721676e"
server_key_version: "encryptionkey@0"
I20260812 02:42:09.007771 8848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:42:09.021347 8848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:42:09.024308 8848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:42:09.024509 8848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:42:09.031689 8848 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.189:46567
I20260812 02:42:09.031977 8959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.189:46567 every 8 connection(s)
I20260812 02:42:09.032713 8848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-1/data/info.pb
I20260812 02:42:09.035625 8961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:42:09.038756 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 8848
I20260812 02:42:09.038861 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-1/wal/instance
I20260812 02:42:09.039122 3582 external_mini_cluster.cc:1428] Setting key 697916a4b04813e378cc48ba5a1a6646
I20260812 02:42:09.039588 8961 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.190" port: 44455 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.127.188@KRBTEST.COM; defaulting to no policy
I20260812 02:42:09.049144 8831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.127.188@KRBTEST.COM" created.
Aug 12 02:42:09 dist-test-slave-rmhg krb5kdc[8687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502528, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.189@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
Aug 12 02:42:09 dist-test-slave-rmhg krb5kdc[8687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502528, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for kudu/127.3.127.189@KRBTEST.COM
I20260812 02:42:09.056712 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.041535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33036 (local address 127.3.127.190:44455)
0812 02:42:09.041694 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.041699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.042549 (+ 850us) server_negotiation.cc:407] Connection header received
0812 02:42:09.043497 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.043501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.043571 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.043695 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.044898 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.045769 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.046946 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.047311 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.047436 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.051833 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:09.051836 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:09.051839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:09.051869 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:09.054552 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.055104 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.055108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.055110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.055191 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.055549 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.055552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.055554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.055750 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:09.055840 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:09.056404 (+ 564us) server_negotiation.cc:299] Negotiation successful
0812 02:42:09.056530 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 02:42:09.057657 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.041886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:44455 (local address 127.0.0.1:33036)
0812 02:42:09.042383 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:09.042435 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 02:42:09.043287 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:42:09.043873 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:42:09.043882 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:42:09.044239 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:42:09.044741 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:09.044755 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.045898 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.045901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:09.046814 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:09.046824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.047462 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.047465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:09.047607 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.048182 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:42:09.048199 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:42:09.051665 (+ 3466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:42:09.054677 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:09.054684 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:09.054704 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:09.055005 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:09.055276 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:09.055279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:09.055281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:09.055455 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:09.055862 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:42:09.055870 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:42:09.056146 (+ 276us) client_negotiation.cc:769] Sending connection context
0812 02:42:09.056333 (+ 187us) client_negotiation.cc:240] Negotiation successful
0812 02:42:09.056674 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":131,"threads_started":1}
I20260812 02:42:09.059355 8961 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:42:09.059633 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.049626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:46567 (local address 127.0.0.1:51584)
0812 02:42:09.049776 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:09.049789 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:42:09.049857 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:42:09.050526 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:42:09.050529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:42:09.050723 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:42:09.050853 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:09.050860 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.052122 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.052125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:09.052773 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:09.052784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.053599 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.053603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:09.053699 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.054116 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:42:09.054132 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:42:09.055705 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:42:09.058005 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:09.058009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:09.058011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:09.058253 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:09.058514 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:09.058517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:09.058519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:09.058617 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:09.059110 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:42:09.059113 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:42:09.059227 (+ 114us) client_negotiation.cc:769] Sending connection context
0812 02:42:09.059336 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 02:42:09.059478 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260812 02:42:09.060321 8966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.049749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51584 (local address 127.3.127.189:46567)
0812 02:42:09.050114 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.050123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.050143 (+ 20us) server_negotiation.cc:407] Connection header received
0812 02:42:09.050179 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.050183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.050343 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.050501 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.051002 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.052014 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.052905 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.053484 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.053591 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.055863 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:09.055874 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:09.055889 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:09.055914 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:09.057904 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.058345 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.058351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.058353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.058433 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.058713 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.058717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.058718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.059005 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:09.059129 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:09.059997 (+ 868us) server_negotiation.cc:299] Negotiation successful
0812 02:42:09.060155 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":174,"threads_started":1}
I20260812 02:42:09.062556 8831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:42:09.063390 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.063065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:45891 (local address 127.0.0.1:59550)
0812 02:42:09.063254 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:09.063314 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:45891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260812 02:42:09.064095 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.063857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:45891 (local address 127.0.0.1:59558)
0812 02:42:09.063995 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:09.064042 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:45891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260812 02:42:09.064256 8831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.188:45891: Network error: Client connection negotiation failed: client connection to 127.3.127.188:45891: connect: Connection refused (error 111)
Aug 12 02:42:09 dist-test-slave-rmhg krb5kdc[8687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502528, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.189@KRBTEST.COM for kudu/127.3.127.189@KRBTEST.COM
I20260812 02:42:09.068785 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.059858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:46567 (local address 127.0.0.1:51596)
0812 02:42:09.059957 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:09.059970 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:42:09.060041 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:42:09.060545 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:42:09.060548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:42:09.060712 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:42:09.060796 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:09.060801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.061636 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.061639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:09.062372 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:09.062382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.062933 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.062937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:09.063060 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.063666 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:42:09.063678 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:42:09.064974 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:42:09.067222 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:09.067233 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:09.067235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:09.067446 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:09.067814 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:09.067818 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:09.067819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:09.067922 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:09.068309 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:42:09.068312 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:42:09.068427 (+ 115us) client_negotiation.cc:769] Sending connection context
0812 02:42:09.068534 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 02:42:09.068664 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 02:42:09.068971 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.060054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.3.127.189:46567)
0812 02:42:09.060357 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.060364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.060374 (+ 10us) server_negotiation.cc:407] Connection header received
0812 02:42:09.060401 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.060403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.060434 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.060506 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.060897 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.061528 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.062486 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.062795 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.062895 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.065171 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:09.065176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:09.065178 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:09.065205 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:09.067083 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.067584 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.067587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.067588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.067696 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.068012 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.068014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.068015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.068208 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:09.068291 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:09.068567 (+ 276us) server_negotiation.cc:299] Negotiation successful
0812 02:42:09.068776 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
I20260812 02:42:09.069625 8961 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:42:09.070344 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.070052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:45891 (local address 127.0.0.1:59574)
0812 02:42:09.070181 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:09.070276 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:45891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260812 02:42:09.070768 8881 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.127.188:45891: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.127.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.188.keytab.
Entry for principal kudu/127.3.127.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.188.keytab.
I20260812 02:42:09.072381 3582 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rbVbc/build/debug/bin/kudu
/tmp/dist-test-task9rbVbc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.188:45891
--webserver_interface=127.3.127.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.188.keytab
--principal=kudu/127.3.127.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.127.148:44457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.127.190:44455,127.3.127.189:46567,127.3.127.188:45891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:42:09.072435 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.072221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:45891 (local address 127.0.0.1:59580)
0812 02:42:09.072335 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:09.072382 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:45891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 02:42:09.072654 8961 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.188:45891: Network error: Client connection negotiation failed: client connection to 127.3.127.188:45891: connect: Connection refused (error 111)
I20260812 02:42:09.095433 8831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } attempt: 1
I20260812 02:42:09.096369 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.096011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:45891 (local address 127.0.0.1:59594)
0812 02:42:09.096205 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:09.096276 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:45891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 02:42:09.097134 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.096901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:45891 (local address 127.0.0.1:59596)
0812 02:42:09.097016 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:09.097067 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:45891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260812 02:42:09.097304 8831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.188:45891: Network error: Client connection negotiation failed: client connection to 127.3.127.188:45891: connect: Connection refused (error 111)
I20260812 02:42:09.121939 8961 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } attempt: 1
I20260812 02:42:09.122864 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.122549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:45891 (local address 127.0.0.1:59604)
0812 02:42:09.122718 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:09.122782 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:45891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 02:42:09.123579 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.123356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:45891 (local address 127.0.0.1:59608)
0812 02:42:09.123465 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:09.123516 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:45891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 02:42:09.123705 8961 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.188:45891: Network error: Client connection negotiation failed: client connection to 127.3.127.188:45891: connect: Connection refused (error 111)
I20260812 02:42:09.172482 8831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } attempt: 2
I20260812 02:42:09.173491 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.173141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:45891 (local address 127.0.0.1:59612)
0812 02:42:09.173331 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:09.173401 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:45891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 02:42:09.174216 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.173996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:45891 (local address 127.0.0.1:59616)
0812 02:42:09.174103 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:09.174151 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:45891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260812 02:42:09.174333 8831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.188:45891: Network error: Client connection negotiation failed: client connection to 127.3.127.188:45891: connect: Connection refused (error 111)
W20260812 02:42:09.191573 8972 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 02:42:09.191869 8972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:42:09.191915 8972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:42:09.191978 8972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 02:42:09.191870 8961 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } attempt: 2
I20260812 02:42:09.192713 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.192379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:45891 (local address 127.0.0.1:59618)
0812 02:42:09.192568 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:09.192630 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:45891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 02:42:09.193423 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.193219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:45891 (local address 127.0.0.1:59634)
0812 02:42:09.193316 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:09.193363 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.127.188:45891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 02:42:09.193534 8961 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.127.188:45891: Network error: Client connection negotiation failed: client connection to 127.3.127.188:45891: connect: Connection refused (error 111)
W20260812 02:42:09.196190 8972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:42:09.196244 8972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:42:09.196271 8972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:42:09.196300 8972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:42:09.196328 8972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:42:09.201674 8972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.127.148:44457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.127.190:44455,127.3.127.189:46567,127.3.127.188:45891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/krb5kdc/kudu_127.3.127.188.keytab
--principal=kudu/127.3.127.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.127.188:45891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.127.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:42:09.202765 8972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:42:09.203786 8972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:42:09.212442 8983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:42:09.212425 8984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:42:09.212425 8986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:42:09.212520 8972 server_base.cc:1034] running on GCE node
I20260812 02:42:09.213482 8972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:42:09.214634 8972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:42:09.215827 8972 hybrid_clock.cc:648] HybridClock initialized: now 1786502529215802 us; error 34 us; skew 500 ppm
Aug 12 02:42:09 dist-test-slave-rmhg krb5kdc[8687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502529, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:42:09.218796 8972 init.cc:377] Logged in from keytab as kudu/127.3.127.188@KRBTEST.COM (short username kudu)
I20260812 02:42:09.219807 8972 webserver.cc:460] Webserver started at http://127.3.127.188:34923/ using document root <none> and password file <none>
I20260812 02:42:09.220391 8972 fs_manager.cc:359] Metadata directory not provided
I20260812 02:42:09.220461 8972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:42:09.220691 8972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:42:09.222957 8972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-2/data/instance:
uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519"
format_stamp: "Formatted at 2026-08-12 02:42:09 on dist-test-slave-rmhg"
server_key: "ca76b3449ae17bcb19a90ac1a2539855"
server_key_iv: "66d8fa9e41b998a48120687c9694d84c"
server_key_version: "encryptionkey@0"
I20260812 02:42:09.223541 8972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-2/wal/instance:
uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519"
format_stamp: "Formatted at 2026-08-12 02:42:09 on dist-test-slave-rmhg"
server_key: "ca76b3449ae17bcb19a90ac1a2539855"
server_key_iv: "66d8fa9e41b998a48120687c9694d84c"
server_key_version: "encryptionkey@0"
I20260812 02:42:09.227597 8972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260812 02:42:09.230319 8999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:42:09.231664 8972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:42:09.231815 8972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-2/data,/tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-2/wal
uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519"
format_stamp: "Formatted at 2026-08-12 02:42:09 on dist-test-slave-rmhg"
server_key: "ca76b3449ae17bcb19a90ac1a2539855"
server_key_iv: "66d8fa9e41b998a48120687c9694d84c"
server_key_version: "encryptionkey@0"
I20260812 02:42:09.231928 8972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:42:09.246954 8972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:42:09.250380 8972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:42:09.250607 8972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:42:09.258026 8972 rpc_server.cc:307] RPC server started. Bound to: 127.3.127.188:45891
I20260812 02:42:09.258308 9085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.127.188:45891 every 8 connection(s)
I20260812 02:42:09.259066 8972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-2/data/info.pb
I20260812 02:42:09.260131 3582 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rbVbc/build/debug/bin/kudu as pid 8972
I20260812 02:42:09.260265 3582 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rbVbc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786502224378336-3582-0/minicluster-data/master-2/wal/instance
I20260812 02:42:09.260591 3582 external_mini_cluster.cc:1428] Setting key e05c996eb0cb51e1338320eb8879b27f
I20260812 02:42:09.262516 9086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:42:09.266981 9086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.190" port: 44455 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:42:09 dist-test-slave-rmhg krb5kdc[8687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.188@KRBTEST.COM
I20260812 02:42:09.273489 8831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } attempt: 3
Aug 12 02:42:09 dist-test-slave-rmhg krb5kdc[8687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502529, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.188@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:42:09.275805 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.261971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59646 (local address 127.3.127.188:45891)
0812 02:42:09.262412 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.262429 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.262467 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:42:09.263192 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.263210 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.263573 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.263871 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.264853 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.265964 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.267134 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.267670 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.267794 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.270551 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:09.270556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:09.270567 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:09.270595 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:09.272158 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.272568 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.272573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.272578 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.272735 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.273027 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.273029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.273030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.273576 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:09.273768 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:09.274545 (+ 777us) server_negotiation.cc:299] Negotiation successful
0812 02:42:09.274836 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":128,"threads_started":1}
I20260812 02:42:09.279148 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.269007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.190:44455 (local address 127.0.0.1:33044)
0812 02:42:09.269494 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:09.269536 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 02:42:09.269667 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:42:09.269945 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:42:09.269952 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:42:09.270324 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:42:09.270467 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:09.270473 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.271202 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.271205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:09.271836 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:09.271846 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.272276 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.272279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:09.272398 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.272916 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:42:09.272931 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:42:09.275271 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:42:09.277416 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:09.277424 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:09.277429 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:09.277728 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:09.278042 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:09.278045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:09.278047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:09.278131 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:09.278597 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:42:09.278601 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:42:09.278707 (+ 106us) client_negotiation.cc:769] Sending connection context
0812 02:42:09.278814 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 02:42:09.278952 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":188,"threads_started":1}
I20260812 02:42:09.279196 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.269076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33044 (local address 127.3.127.190:44455)
0812 02:42:09.269265 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.269270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.269613 (+ 343us) server_negotiation.cc:407] Connection header received
0812 02:42:09.269760 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.269763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.269810 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.269876 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.270594 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.271087 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.271952 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.272174 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.272263 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.275432 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:09.275436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:09.275439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:09.275468 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:09.277299 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.277843 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.277846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.277848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.277938 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.278231 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.278235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.278236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.278469 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:09.278531 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:09.278912 (+ 381us) server_negotiation.cc:299] Negotiation successful
0812 02:42:09.279045 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 02:42:09.281013 9086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:42:09 dist-test-slave-rmhg krb5kdc[8687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502528, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.190@KRBTEST.COM for kudu/127.3.127.188@KRBTEST.COM
I20260812 02:42:09.284675 8961 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } attempt: 3
I20260812 02:42:09.285430 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.274461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:45891 (local address 127.0.0.1:59648)
0812 02:42:09.274695 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:09.274712 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:42:09.274794 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:42:09.276339 (+ 1545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:42:09.276375 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:42:09.276810 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:42:09.276930 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:09.276938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.279061 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.279064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:09.279788 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:09.279798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.280584 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.280587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:09.280681 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.281145 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:42:09.281192 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:42:09.282501 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:42:09.284024 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:09.284027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:09.284030 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:09.284224 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:09.284479 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:09.284481 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:09.284483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:09.284563 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:09.284911 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:42:09.284914 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:42:09.285008 (+ 94us) client_negotiation.cc:769] Sending connection context
0812 02:42:09.285108 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 02:42:09.285265 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
Aug 12 02:42:09 dist-test-slave-rmhg krb5kdc[8687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.190@KRBTEST.COM
I20260812 02:42:09.286660 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.274100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59648 (local address 127.3.127.188:45891)
0812 02:42:09.276149 (+ 2049us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.276153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.276166 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:42:09.276197 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.276199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.276236 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.276302 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.277073 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.278212 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.280147 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.280446 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.280590 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.282623 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:09.282626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:09.282629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:09.282654 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:09.283922 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.284313 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.284316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.284318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.284388 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.284645 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.284649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.284650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.284826 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:09.284918 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:09.286360 (+ 1442us) server_negotiation.cc:299] Negotiation successful
0812 02:42:09.286480 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1973,"thread_start_us":356,"threads_started":1}
Aug 12 02:42:09 dist-test-slave-rmhg krb5kdc[8687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502529, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.188@KRBTEST.COM for kudu/127.3.127.189@KRBTEST.COM
Aug 12 02:42:09 dist-test-slave-rmhg krb5kdc[8687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502528, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.189@KRBTEST.COM for kudu/127.3.127.188@KRBTEST.COM
I20260812 02:42:09.290452 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.279870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33050 (local address 127.3.127.190:44455)
0812 02:42:09.279978 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.279982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.279994 (+ 12us) server_negotiation.cc:407] Connection header received
0812 02:42:09.280146 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.280149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.280200 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.280263 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.280958 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.281613 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.282622 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.282933 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.283003 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.286852 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:09.286857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:09.286859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:09.286887 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:09.288384 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.288875 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.288879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.288881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.288970 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.289353 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.289357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.289358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.289574 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:09.289767 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:09.290174 (+ 407us) server_negotiation.cc:299] Negotiation successful
0812 02:42:09.290301 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 02:42:09.291286 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.281772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.3.127.189:46567)
0812 02:42:09.281906 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.281910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.281925 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:42:09.281969 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.281972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.282022 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.282107 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.282556 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.283281 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.284980 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.285316 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.285383 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.287575 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:09.287579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:09.287582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:09.287608 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:09.289434 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.289961 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.289964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.289966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.290056 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.290364 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.290368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.290370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.290572 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:09.290645 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:09.291011 (+ 366us) server_negotiation.cc:299] Negotiation successful
0812 02:42:09.291134 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 02:42:09.291452 8831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8: Bootstrap starting.
I20260812 02:42:09.291772 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.281498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.189:46567 (local address 127.0.0.1:51604)
0812 02:42:09.281768 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:09.281781 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:42:09.281845 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:42:09.282131 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:42:09.282134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:42:09.282301 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:42:09.282411 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:09.282417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.283417 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.283420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:09.284846 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:09.284856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.285528 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.285531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:09.285646 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.286059 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:42:09.286073 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:42:09.287415 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:42:09.289543 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:09.289546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:09.289548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:09.289857 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:09.290169 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:09.290172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:09.290173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:09.290258 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:09.290729 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:42:09.290732 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:42:09.290825 (+ 93us) client_negotiation.cc:769] Sending connection context
0812 02:42:09.291475 (+ 650us) client_negotiation.cc:240] Negotiation successful
0812 02:42:09.291614 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198}
I20260812 02:42:09.292416 9086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:42:09.294942 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.285083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:45891 (local address 127.0.0.1:59654)
0812 02:42:09.285276 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:09.285290 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:42:09.285365 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:42:09.285647 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:42:09.285650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:42:09.285828 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:42:09.285929 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:09.285936 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.287016 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.287019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:09.287787 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:09.287799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.288395 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.288398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:09.288485 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.289267 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:42:09.289281 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:42:09.290987 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:42:09.293409 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:09.293412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:09.293415 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:09.293674 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:09.293976 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:09.293979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:09.293981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:09.294064 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:09.294477 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:42:09.294480 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:42:09.294577 (+ 97us) client_negotiation.cc:769] Sending connection context
0812 02:42:09.294669 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 02:42:09.294793 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260812 02:42:09.295243 8831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8: Neither blocks nor log segments found. Creating new log.
I20260812 02:42:09.295768 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.285188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59654 (local address 127.3.127.188:45891)
0812 02:42:09.285295 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.285307 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.285322 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:42:09.285444 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.285447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.285493 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.286039 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.286146 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.286887 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.287932 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.288266 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.288365 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.291140 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:09.291144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:09.291146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:09.291170 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:09.293292 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.293796 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.293800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.293801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.293882 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.294170 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.294173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.294174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.294366 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:09.295369 (+ 1003us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:09.295469 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 02:42:09.295591 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 02:42:09.296466 8831 log.cc:824] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:42:09 dist-test-slave-rmhg krb5kdc[8687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502529, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.127.188@KRBTEST.COM for kudu/127.3.127.188@KRBTEST.COM
Aug 12 02:42:09 dist-test-slave-rmhg krb5kdc[8687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.127.189@KRBTEST.COM
I20260812 02:42:09.298907 8831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8: No bootstrap required, opened a new log
I20260812 02:42:09.299214 8961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77a8c0c3fb4a43c58a30ba4e53214643: Bootstrap starting.
I20260812 02:42:09.301836 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.292908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.127.188:45891 (local address 127.0.0.1:59670)
0812 02:42:09.293091 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 02:42:09.293103 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:42:09.293191 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:42:09.293586 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:42:09.293588 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:42:09.293755 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:42:09.293836 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:09.293843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.294714 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.294717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:09.295485 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:42:09.295496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.296107 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.296110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:42:09.296180 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.296642 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:42:09.296657 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:42:09.298160 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:42:09.300137 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:09.300140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:09.300142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:09.300369 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:09.300644 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:42:09.300647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:42:09.300648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:42:09.300736 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:42:09.301298 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:42:09.301302 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:42:09.301401 (+ 99us) client_negotiation.cc:769] Sending connection context
0812 02:42:09.301508 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 02:42:09.301649 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":117,"threads_started":1}
I20260812 02:42:09.302068 8961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77a8c0c3fb4a43c58a30ba4e53214643: Neither blocks nor log segments found. Creating new log.
I20260812 02:42:09.302323 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.293268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59670 (local address 127.3.127.188:45891)
0812 02:42:09.293364 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.293368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.293378 (+ 10us) server_negotiation.cc:407] Connection header received
0812 02:42:09.293405 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.293407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.293442 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.293503 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.293939 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.294608 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.295600 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.295925 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.296008 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.298299 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:09.298303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:09.298306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:09.298327 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:09.300029 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.300471 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.300474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.300476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.300552 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.300824 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.300826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.300828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.301033 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:09.301116 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:09.302081 (+ 965us) server_negotiation.cc:299] Negotiation successful
0812 02:42:09.302188 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 02:42:09.303059 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.292133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51610 (local address 127.3.127.189:46567)
0812 02:42:09.292788 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.292791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.292801 (+ 10us) server_negotiation.cc:407] Connection header received
0812 02:42:09.292829 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.292831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.292863 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.292927 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.293717 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.294472 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.295661 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.295978 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.296040 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.299516 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:09.299521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:09.299523 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:09.299551 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:09.301075 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.301585 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.301589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.301590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.301675 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.301987 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.301991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.301992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.302218 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:09.302386 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:09.302803 (+ 417us) server_negotiation.cc:299] Negotiation successful
0812 02:42:09.302914 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589}
I20260812 02:42:09.303432 8831 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0d6199a40d4f969f124ef417b356a8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 44455 } } peers { permanent_uuid: "77a8c0c3fb4a43c58a30ba4e53214643" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } } peers { permanent_uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } }
I20260812 02:42:09.303695 8831 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd0d6199a40d4f969f124ef417b356a8, State: Initialized, Role: FOLLOWER
I20260812 02:42:09.304039 8831 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0d6199a40d4f969f124ef417b356a8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 44455 } } peers { permanent_uuid: "77a8c0c3fb4a43c58a30ba4e53214643" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } } peers { permanent_uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } }
I20260812 02:42:09.304633 9096 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0d6199a40d4f969f124ef417b356a8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 44455 } } peers { permanent_uuid: "77a8c0c3fb4a43c58a30ba4e53214643" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } } peers { permanent_uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } } }
I20260812 02:42:09.304986 9096 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:42:09.305625 8831 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:42:09.305716 9086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae9a5d0aeb44fa3a8b62c890ebc9519: Bootstrap starting.
I20260812 02:42:09.306321 8961 log.cc:824] T 00000000000000000000000000000000 P 77a8c0c3fb4a43c58a30ba4e53214643: Log is configured to *not* fsync() on all Append() calls
I20260812 02:42:09.306455 3582 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:42:09.307638 8783 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bd0d6199a40d4f969f124ef417b356a8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33050
I20260812 02:42:09.307858 8783 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 02:42:09.308152 8783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:42:09.308576 9086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cae9a5d0aeb44fa3a8b62c890ebc9519: Neither blocks nor log segments found. Creating new log.
I20260812 02:42:09.309434 9086 log.cc:824] T 00000000000000000000000000000000 P cae9a5d0aeb44fa3a8b62c890ebc9519: Log is configured to *not* fsync() on all Append() calls
I20260812 02:42:09.309510 8783 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0d6199a40d4f969f124ef417b356a8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 44455 } } peers { permanent_uuid: "77a8c0c3fb4a43c58a30ba4e53214643" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } } peers { permanent_uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } }
I20260812 02:42:09.310568 8783 leader_election.cc:288] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [CANDIDATE]: Term 1 election: Requested vote from peers 77a8c0c3fb4a43c58a30ba4e53214643 (127.3.127.189:46567), cae9a5d0aeb44fa3a8b62c890ebc9519 (127.3.127.188:45891)
I20260812 02:42:09.311265 8961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77a8c0c3fb4a43c58a30ba4e53214643: No bootstrap required, opened a new log
I20260812 02:42:09.311833 9086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae9a5d0aeb44fa3a8b62c890ebc9519: No bootstrap required, opened a new log
I20260812 02:42:09.315778 8961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77a8c0c3fb4a43c58a30ba4e53214643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0d6199a40d4f969f124ef417b356a8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 44455 } } peers { permanent_uuid: "77a8c0c3fb4a43c58a30ba4e53214643" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } } peers { permanent_uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } }
I20260812 02:42:09.315852 9086 raft_consensus.cc:348] T 00000000000000000000000000000000 P cae9a5d0aeb44fa3a8b62c890ebc9519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0d6199a40d4f969f124ef417b356a8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 44455 } } peers { permanent_uuid: "77a8c0c3fb4a43c58a30ba4e53214643" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } } peers { permanent_uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } }
I20260812 02:42:09.316104 9086 raft_consensus.cc:729] T 00000000000000000000000000000000 P cae9a5d0aeb44fa3a8b62c890ebc9519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae9a5d0aeb44fa3a8b62c890ebc9519, State: Initialized, Role: FOLLOWER
I20260812 02:42:09.316149 8961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77a8c0c3fb4a43c58a30ba4e53214643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77a8c0c3fb4a43c58a30ba4e53214643, State: Initialized, Role: FOLLOWER
I20260812 02:42:09.316461 9086 consensus_queue.cc:260] T 00000000000000000000000000000000 P cae9a5d0aeb44fa3a8b62c890ebc9519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0d6199a40d4f969f124ef417b356a8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 44455 } } peers { permanent_uuid: "77a8c0c3fb4a43c58a30ba4e53214643" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } } peers { permanent_uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } }
I20260812 02:42:09.316558 8961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77a8c0c3fb4a43c58a30ba4e53214643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0d6199a40d4f969f124ef417b356a8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 44455 } } peers { permanent_uuid: "77a8c0c3fb4a43c58a30ba4e53214643" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } } peers { permanent_uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } }
I20260812 02:42:09.317044 9129 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae9a5d0aeb44fa3a8b62c890ebc9519 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0d6199a40d4f969f124ef417b356a8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 44455 } } peers { permanent_uuid: "77a8c0c3fb4a43c58a30ba4e53214643" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } } peers { permanent_uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } } }
I20260812 02:42:09.317358 9129 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae9a5d0aeb44fa3a8b62c890ebc9519 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:42:09.318005 9086 sys_catalog.cc:564] T 00000000000000000000000000000000 P cae9a5d0aeb44fa3a8b62c890ebc9519 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:42:09.318048 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd0d6199a40d4f969f124ef417b356a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519"
I20260812 02:42:09.318269 9039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cae9a5d0aeb44fa3a8b62c890ebc9519 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:42:09.318423 8961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77a8c0c3fb4a43c58a30ba4e53214643 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:42:09.315297 8711 master_runner.cc:186] Error getting master registration for 127.3.127.189:46567: OK, instance_id { permanent_uuid: "77a8c0c3fb4a43c58a30ba4e53214643" instance_seqno: 1786502529029657 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 02:42:09.319195 8711 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 02:42:09.320240 9039 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cae9a5d0aeb44fa3a8b62c890ebc9519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd0d6199a40d4f969f124ef417b356a8 in term 1.
I20260812 02:42:09.320550 8920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd0d6199a40d4f969f124ef417b356a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "77a8c0c3fb4a43c58a30ba4e53214643"
I20260812 02:42:09.320809 8920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77a8c0c3fb4a43c58a30ba4e53214643 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:42:09.321012 8740 leader_election.cc:302] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd0d6199a40d4f969f124ef417b356a8, cae9a5d0aeb44fa3a8b62c890ebc9519; no voters:
I20260812 02:42:09.321396 9096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:42:09.321619 9096 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [term 1 LEADER]: Becoming Leader. State: Replica: bd0d6199a40d4f969f124ef417b356a8, State: Running, Role: LEADER
I20260812 02:42:09.321975 9096 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0d6199a40d4f969f124ef417b356a8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 44455 } } peers { permanent_uuid: "77a8c0c3fb4a43c58a30ba4e53214643" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } } peers { permanent_uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } }
I20260812 02:42:09.322265 8920 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 77a8c0c3fb4a43c58a30ba4e53214643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd0d6199a40d4f969f124ef417b356a8 in term 1.
W20260812 02:42:09.322763 9112 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:42:09.322842 9112 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 02:42:09.325975 9180 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 77a8c0c3fb4a43c58a30ba4e53214643: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:42:09.326054 9180 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:42:09.328190 9130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77a8c0c3fb4a43c58a30ba4e53214643 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0d6199a40d4f969f124ef417b356a8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 44455 } } peers { permanent_uuid: "77a8c0c3fb4a43c58a30ba4e53214643" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } } peers { permanent_uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } } }
I20260812 02:42:09.328313 9130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77a8c0c3fb4a43c58a30ba4e53214643 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:42:09.335088 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.313015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33056 (local address 127.3.127.190:44455)
0812 02:42:09.313136 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.313140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.320248 (+ 7108us) server_negotiation.cc:407] Connection header received
0812 02:42:09.320476 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.320479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.320537 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.320593 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.324996 (+ 4403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.325767 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.326912 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.327218 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.327292 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.330312 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:09.330316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:09.330320 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:09.330347 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:09.332079 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.332583 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.332587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.332589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.332699 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.333333 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.333336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.333338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.333573 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:09.333723 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:09.334765 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0812 02:42:09.334916 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 02:42:09.328653 8848 master_runner.cc:186] Error getting master registration for 127.3.127.188:45891: OK, instance_id { permanent_uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519" instance_seqno: 1786502529255970 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 02:42:09.336957 8848 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 02:42:09.337373 9179 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd0d6199a40d4f969f124ef417b356a8. Latest consensus state: current_term: 1 leader_uuid: "bd0d6199a40d4f969f124ef417b356a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0d6199a40d4f969f124ef417b356a8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 44455 } } peers { permanent_uuid: "77a8c0c3fb4a43c58a30ba4e53214643" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } } peers { permanent_uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } } }
I20260812 02:42:09.337596 9179 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [sys.catalog]: This master's current role is: LEADER
I20260812 02:42:09.338104 9184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:42:09.339073 9184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260812 02:42:09.339926 9183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cae9a5d0aeb44fa3a8b62c890ebc9519: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:42:09.340015 9183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:42:09.344038 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.315063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.3.127.188:45891)
0812 02:42:09.319376 (+ 4313us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.319381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.331234 (+ 11853us) server_negotiation.cc:407] Connection header received
0812 02:42:09.333070 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.333073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.333125 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.333194 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.336623 (+ 3429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.337357 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.338381 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.338696 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.338767 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.340493 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:09.340498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:09.340501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:09.340527 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:09.342131 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.342634 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.342638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.342640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.342727 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.343035 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.343038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.343040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.343265 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:09.343366 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:09.343748 (+ 382us) server_negotiation.cc:299] Negotiation successful
0812 02:42:09.343871 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4216}
I20260812 02:42:09.345042 9039 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cae9a5d0aeb44fa3a8b62c890ebc9519 [term 1 FOLLOWER]: Refusing update from remote peer bd0d6199a40d4f969f124ef417b356a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:42:09.345342 8920 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 77a8c0c3fb4a43c58a30ba4e53214643 [term 1 FOLLOWER]: Refusing update from remote peer bd0d6199a40d4f969f124ef417b356a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:42:09.345772 9179 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:42:09.346217 9179 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77a8c0c3fb4a43c58a30ba4e53214643" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:42:09.350245 9130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77a8c0c3fb4a43c58a30ba4e53214643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd0d6199a40d4f969f124ef417b356a8. Latest consensus state: current_term: 1 leader_uuid: "bd0d6199a40d4f969f124ef417b356a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0d6199a40d4f969f124ef417b356a8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 44455 } } peers { permanent_uuid: "77a8c0c3fb4a43c58a30ba4e53214643" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } } peers { permanent_uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } } }
I20260812 02:42:09.350378 9130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77a8c0c3fb4a43c58a30ba4e53214643 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:42:09.354029 9129 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae9a5d0aeb44fa3a8b62c890ebc9519 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd0d6199a40d4f969f124ef417b356a8. Latest consensus state: current_term: 1 leader_uuid: "bd0d6199a40d4f969f124ef417b356a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0d6199a40d4f969f124ef417b356a8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 44455 } } peers { permanent_uuid: "77a8c0c3fb4a43c58a30ba4e53214643" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } } peers { permanent_uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } } }
I20260812 02:42:09.354147 9129 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae9a5d0aeb44fa3a8b62c890ebc9519 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:42:09.355847 9096 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd0d6199a40d4f969f124ef417b356a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0d6199a40d4f969f124ef417b356a8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 44455 } } peers { permanent_uuid: "77a8c0c3fb4a43c58a30ba4e53214643" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } } peers { permanent_uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } } }
I20260812 02:42:09.355978 9096 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [sys.catalog]: This master's current role is: LEADER
I20260812 02:42:09.357124 9192 mvcc.cc:205] Tried to move back new op lower bound from 7317514360188915712 to 7317514360106729472. Current Snapshot: MvccSnapshot[applied={T|T < 7317514360188915712}]
I20260812 02:42:09.357355 9129 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae9a5d0aeb44fa3a8b62c890ebc9519 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd0d6199a40d4f969f124ef417b356a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0d6199a40d4f969f124ef417b356a8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 44455 } } peers { permanent_uuid: "77a8c0c3fb4a43c58a30ba4e53214643" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } } peers { permanent_uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } } }
I20260812 02:42:09.357451 9129 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae9a5d0aeb44fa3a8b62c890ebc9519 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:42:09.359378 9184 catalog_manager.cc:1347] Generated new cluster ID: 4b356feda6cb44a6a6b4419edb91af0d
I20260812 02:42:09.359440 9184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:42:09.370939 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:09.313489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51626 (local address 127.3.127.189:46567)
0812 02:42:09.313591 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.313596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.328760 (+ 15164us) server_negotiation.cc:407] Connection header received
0812 02:42:09.328878 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.328881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.328929 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.328999 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.333952 (+ 4953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.334701 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.335841 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:09.336161 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:09.336241 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:09.350127 (+ 13886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:42:09.350132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:42:09.350135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:42:09.350169 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:42:09.352831 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.369305 (+ 16474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.369310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.369312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.369452 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:42:09.369849 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:42:09.369853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:42:09.369855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:42:09.370089 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:42:09.370207 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:09.370587 (+ 380us) server_negotiation.cc:299] Negotiation successful
0812 02:42:09.370754 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 02:42:09.373713 9184 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:42:09.376187 9130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77a8c0c3fb4a43c58a30ba4e53214643 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd0d6199a40d4f969f124ef417b356a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0d6199a40d4f969f124ef417b356a8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 44455 } } peers { permanent_uuid: "77a8c0c3fb4a43c58a30ba4e53214643" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } } peers { permanent_uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } } }
I20260812 02:42:09.376329 9130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77a8c0c3fb4a43c58a30ba4e53214643 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:42:09.377449 9190 mvcc.cc:205] Tried to move back new op lower bound from 7317514360188915712 to 7317514360106729472. Current Snapshot: MvccSnapshot[applied={T|T < 7317514360188915712}]
I20260812 02:42:09.374475 9179 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd0d6199a40d4f969f124ef417b356a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0d6199a40d4f969f124ef417b356a8" member_type: VOTER last_known_addr { host: "127.3.127.190" port: 44455 } } peers { permanent_uuid: "77a8c0c3fb4a43c58a30ba4e53214643" member_type: VOTER last_known_addr { host: "127.3.127.189" port: 46567 } } peers { permanent_uuid: "cae9a5d0aeb44fa3a8b62c890ebc9519" member_type: VOTER last_known_addr { host: "127.3.127.188" port: 45891 } } }
I20260812 02:42:09.378139 9179 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8 [sys.catalog]: This master's current role is: LEADER
I20260812 02:42:09.379559 9184 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:42:09.399365 9184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd0d6199a40d4f969f124ef417b356a8: Generated new TSK 0
I20260812 02:42:09.400455 9184 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 02:42:09.444658 8968 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:09.440793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51628 (local address 127.3.127.189:46567)
0812 02:42:09.440933 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.440938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.443377 (+ 2439us) server_negotiation.cc:407] Connection header received
0812 02:42:09.443590 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.443594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.443660 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.443716 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.444486 (+ 770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51628: BlockingRecv error: recv got EOF from 127.0.0.1:51628 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260812 02:42:09.446233 8838 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:09.441774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33068 (local address 127.3.127.190:44455)
0812 02:42:09.441892 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.441897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.445256 (+ 3359us) server_negotiation.cc:407] Connection header received
0812 02:42:09.445452 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.445455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.445509 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.445559 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.446094 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33068: BlockingRecv error: recv got EOF from 127.0.0.1:33068 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260812 02:42:09.447747 9091 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:09.442480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59682 (local address 127.3.127.188:45891)
0812 02:42:09.442753 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.442758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.446762 (+ 4004us) server_negotiation.cc:407] Connection header received
0812 02:42:09.446946 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.446949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.447006 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.447057 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.447600 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59682: BlockingRecv error: recv got EOF from 127.0.0.1:59682 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260812 02:42:09.452677 8968 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:09.450966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51638 (local address 127.3.127.189:46567)
0812 02:42:09.451082 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.451086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.451569 (+ 483us) server_negotiation.cc:407] Connection header received
0812 02:42:09.451824 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.451828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.451914 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.451967 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.452549 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51638: BlockingRecv error: recv got EOF from 127.0.0.1:51638 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260812 02:42:09.458149 8968 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:09.456430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.3.127.189:46567)
0812 02:42:09.456540 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.456544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.457089 (+ 545us) server_negotiation.cc:407] Connection header received
0812 02:42:09.457350 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.457354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.457408 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.457459 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.458029 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51654: BlockingRecv error: recv got EOF from 127.0.0.1:51654 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260812 02:42:09.464279 8968 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:09.462475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.3.127.189:46567)
0812 02:42:09.462600 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.462604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.463207 (+ 603us) server_negotiation.cc:407] Connection header received
0812 02:42:09.463426 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.463430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.463489 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.463542 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.464149 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51656: BlockingRecv error: recv got EOF from 127.0.0.1:51656 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260812 02:42:09.472898 8968 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:09.470827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51672 (local address 127.3.127.189:46567)
0812 02:42:09.471003 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.471008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.471716 (+ 708us) server_negotiation.cc:407] Connection header received
0812 02:42:09.471960 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.471964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.472027 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.472087 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.472752 (+ 665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51672: BlockingRecv error: recv got EOF from 127.0.0.1:51672 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260812 02:42:09.485750 8968 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:09.483623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51682 (local address 127.3.127.189:46567)
0812 02:42:09.483789 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.483793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.484524 (+ 731us) server_negotiation.cc:407] Connection header received
0812 02:42:09.484787 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.484791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.484851 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.484910 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.485602 (+ 692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51682: BlockingRecv error: recv got EOF from 127.0.0.1:51682 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260812 02:42:09.507246 8968 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:09.504967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51688 (local address 127.3.127.189:46567)
0812 02:42:09.505121 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.505126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.505916 (+ 790us) server_negotiation.cc:407] Connection header received
0812 02:42:09.506189 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.506195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.506266 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.506331 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.507110 (+ 779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51688: BlockingRecv error: recv got EOF from 127.0.0.1:51688 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260812 02:42:09.543553 8968 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:09.541988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.3.127.189:46567)
0812 02:42:09.542123 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.542128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.542410 (+ 282us) server_negotiation.cc:407] Connection header received
0812 02:42:09.542555 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.542557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.542620 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.542694 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.543408 (+ 714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51696: BlockingRecv error: recv got EOF from 127.0.0.1:51696 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260812 02:42:09.610757 8968 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:09.609631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.3.127.189:46567)
0812 02:42:09.609785 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.609789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.609911 (+ 122us) server_negotiation.cc:407] Connection header received
0812 02:42:09.610076 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.610080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.610145 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.610239 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.610649 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51698: BlockingRecv error: recv got EOF from 127.0.0.1:51698 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260812 02:42:09.742204 8968 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:09.740956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45354 (local address 127.3.127.189:46567)
0812 02:42:09.741183 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:09.741188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:09.741283 (+ 95us) server_negotiation.cc:407] Connection header received
0812 02:42:09.741469 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:09.741473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:09.741536 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:09.741618 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:09.742072 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45354: BlockingRecv error: recv got EOF from 127.0.0.1:45354 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260812 02:42:10.001879 8968 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:42:10.000691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45368 (local address 127.3.127.189:46567)
0812 02:42:10.000845 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:10.000850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:10.000901 (+ 51us) server_negotiation.cc:407] Connection header received
0812 02:42:10.001085 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:10.001089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:10.001139 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:10.001238 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:42:10.001751 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45368: BlockingRecv error: recv got EOF from 127.0.0.1:45368 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 02:42:10.327442 9180 catalog_manager.cc:1259] Loaded cluster ID: 4b356feda6cb44a6a6b4419edb91af0d
I20260812 02:42:10.327540 9180 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 77a8c0c3fb4a43c58a30ba4e53214643: loading cluster ID for follower catalog manager: success
I20260812 02:42:10.329582 9180 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 77a8c0c3fb4a43c58a30ba4e53214643: acquiring CA information for follower catalog manager: success
I20260812 02:42:10.331003 9180 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 77a8c0c3fb4a43c58a30ba4e53214643: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:42:10.341501 9183 catalog_manager.cc:1259] Loaded cluster ID: 4b356feda6cb44a6a6b4419edb91af0d
I20260812 02:42:10.341590 9183 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cae9a5d0aeb44fa3a8b62c890ebc9519: loading cluster ID for follower catalog manager: success
I20260812 02:42:10.343339 9183 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cae9a5d0aeb44fa3a8b62c890ebc9519: acquiring CA information for follower catalog manager: success
I20260812 02:42:10.344696 9183 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cae9a5d0aeb44fa3a8b62c890ebc9519: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:42:10.520582 9298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:10.516156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45370 (local address 127.3.127.189:46567)
0812 02:42:10.516443 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:10.516446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:10.516464 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:42:10.516665 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:10.516669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:10.516816 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:10.516924 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:42:10.517340 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:10.517939 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:10.518971 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:10.519222 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:10.519301 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:10.519622 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:42:10.520196 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:42:10.520291 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:10.520364 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 02:42:10.520419 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":91,"threads_started":1}
I20260812 02:42:10.527441 9308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:10.523704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.3.127.188:45891)
0812 02:42:10.523958 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:10.523963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:10.523976 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:42:10.524072 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:10.524074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:10.524193 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:10.524296 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:42:10.524523 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:10.525052 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:10.525937 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:10.526176 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:10.526266 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:10.526568 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:42:10.527019 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:42:10.527134 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:10.527218 (+ 84us) server_negotiation.cc:299] Negotiation successful
0812 02:42:10.527281 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":67,"threads_started":1}
I20260812 02:42:10.534575 9298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:10.530391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45380 (local address 127.3.127.189:46567)
0812 02:42:10.530611 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:10.530616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:10.530854 (+ 238us) server_negotiation.cc:407] Connection header received
0812 02:42:10.531061 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:10.531064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:10.531118 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:10.531202 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:42:10.531504 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:10.532186 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:10.533239 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:10.533601 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:10.533704 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:10.533947 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:42:10.534046 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:42:10.534128 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:10.534358 (+ 230us) server_negotiation.cc:299] Negotiation successful
0812 02:42:10.534425 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260812 02:42:10.535305 9308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:10.530780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44622 (local address 127.3.127.188:45891)
0812 02:42:10.530892 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:10.530896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:10.531482 (+ 586us) server_negotiation.cc:407] Connection header received
0812 02:42:10.531685 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:10.531688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:10.531733 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:10.531787 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:42:10.532001 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:10.532626 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:10.533808 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:10.534113 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:10.534379 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:10.534791 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:42:10.534905 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:42:10.534954 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:10.535114 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 02:42:10.535180 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 02:42:10.535917 9320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:42:10.530371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40102 (local address 127.3.127.190:44455)
0812 02:42:10.530640 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0812 02:42:10.530644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:42:10.530845 (+ 201us) server_negotiation.cc:407] Connection header received
0812 02:42:10.530946 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:42:10.530949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:42:10.531113 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:42:10.531234 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:42:10.532303 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:10.532925 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:10.533863 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:42:10.534149 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:42:10.534339 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:42:10.534821 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:42:10.535210 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:42:10.535299 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 02:42:10.535712 (+ 413us) server_negotiation.cc:299] Negotiation successful
0812 02:42:10.535779 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260812 02:42:10.538839 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 8711
I20260812 02:42:10.555272 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 8848
I20260812 02:42:10.570680 3582 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rbVbc/build/debug/bin/kudu with pid 8972
2026-08-12T02:42:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3239 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6565 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (306141 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST